Bug 2228186

Summary:	Neutron workers may not have OVN IDLs instantiated
Product:	Red Hat OpenStack	Reporter:	Jakub Libosvar <jlibosva>
Component:	openstack-neutron	Assignee:	Jakub Libosvar <jlibosva>
Status:	ON_DEV ---	QA Contact:	Eran Kuris <ekuris>
Severity:	medium	Docs Contact:
Priority:	high
Version:	17.1 (Wallaby)	CC:	arcsingh, chrisw, kthakre, pgrist, scohen
Target Milestone:	z1	Keywords:	Triaged
Target Release:	17.1	Flags:	ifrangs: needinfo? (jlibosva)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jakub Libosvar 2023-08-01 17:05:32 UTC

Description of problem:
In a very rare occasion during the Neutron server startup, it may happen that connection to MySQL server is lost. If this happens after the parent process spawns workers but before OVN IDLs are instantiated, then setting up hash ring raises an exception when attempting to delete the hash ring node in post_fork_initialize() [1]. This happens before the OVN IDLs are instantiated [2] but after the post fork event is cleared [3].

If this happens, the API worker starts handling requests anyways without having connection to the OVN databases. Any request, e.g. "router list", will hang until HTTP timeout because we wait indefinitely for the post_fork_event to be set [4]. As Neutron is waiting, there is no sign of any ERROR or traceback in the logs and it's not intuitive at all what is happening, as haproxy successfully redirects traffic to Neutron server but there is no sign about the request in Neutron logs.

[1] https://github.com/openstack/neutron/blob/45cdd90640160a78329bc1677b587793c175db48/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py#L345
[2] https://github.com/openstack/neutron/blob/45cdd90640160a78329bc1677b587793c175db48/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py#L348
[3] https://github.com/openstack/neutron/blob/45cdd90640160a78329bc1677b587793c175db48/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py#L341
[4] https://github.com/openstack/neutron/blob/45cdd90640160a78329bc1677b587793c175db48/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py#L173

Version-Release number of selected component (if applicable):
Probably all versions

How reproducible:
Very rare

Steps to Reproduce:
1. Start Neutron server while stopping MySQL DB server or interrupt traffic to the server
2. The MySQL connection must be lost after api workers are spawned but before they clear hash ring nodes.

Actual results:
Neutron stops handling requests that require OVN DB data

Expected results:
Neutron waits on startup until mysql is established

Additional info:
Some requests that don't use OVN DB still work. Only the particular worker that suffered the problem during the startup won't be able to handle requests. For user it leads to very sporadic failures as it requires a request that needs OVN DB and lands on the malfunctioned worker.

Log snippet from the worker start:
2023-08-01 09:33:02.879 7 INFO oslo_service.service [req-b96f1dc7-0ace-4235-b7eb-5a1cafec2a2f - - - - -] Starting 2 workers
2023-08-01 09:33:02.887 7 DEBUG oslo_service.service [req-b96f1dc7-0ace-4235-b7eb-5a1cafec2a2f - - - - -] Started child 105 _start_child /usr/lib/python3.9/site-packages/oslo_service/service.py:576
2023-08-01 09:33:02.893 105 DEBUG neutron_lib.callbacks.manager [-] Notify callbacks ['neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-1809925'] for process, after_
init _notify_loop /usr/lib/python3.9/site-packages/neutron_lib/callbacks/manager.py:192
2023-08-01 09:33:02.896 105 DEBUG oslo_concurrency.lockutils [-] Lock "hash_ring_probe_lock" acquired by "neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._setup_hash_ring" :: waited 0.0
01s inner /usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py:355
2023-08-01 09:33:02.914 105 DEBUG oslo_concurrency.lockutils [req-2d2a4a3d-4132-41c2-a6cb-ccba7c69cf61 - - - - -] Lock "hash_ring_probe_lock" released by "neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._setup_hash_ring" :: held 0.018s inner /usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py:367
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager [req-2d2a4a3d-4132-41c2-a6cb-ccba7c69cf61 - - - - -] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-1809925 process, after_init: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     return fn()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/base.py", line 364, in connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     return _ConnectionFairy._checkout(self)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     fairy = _ConnectionRecord.checkout(pool)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     rec = pool._do_get()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     self._dec_overflow()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     compat.raise_(
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     raise exception
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     return self._create_connection()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     return _ConnectionRecord(self)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     self.__connect(first_connect_check=True)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     pool.logger.debug("Error on connect(): %s", e)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     compat.raise_(
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     raise exception
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     connection = pool._invoke_creator(self)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     return dialect.connect(*cargs, **cparams)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 509, in connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     return self.dbapi.connect(*cargs, **cparams)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/pymysql/__init__.py", line 94, in Connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     return Connection(*args, **kwargs)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 327, in __init__
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     self.connect()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 587, in connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     self._get_server_information()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 969, in _get_server_information
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     packet = self._read_packet()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 646, in _read_packet
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     packet_header = self._read_bytes(4)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 698, in _read_bytes
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     raise err.OperationalError(
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager The above exception was the direct cause of the following exception:
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, **kwargs)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 322, in post_fork_initialize
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     self._setup_hash_ring()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     return f(*args, **kwargs)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 298, in _setup_hash_ring
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     self._clean_hash_ring()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 272, in _clean_hash_ring
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     ovn_hash_ring_db.remove_nodes_from_host(admin_context,
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/neutron/db/ovn_hash_ring_db.py", line 42, in remove_nodes_from_host
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     context.session.query(ovn_models.OVNHashRing).filter(
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 3926, in delete
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     delete_op.exec_()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1697, in exec_
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     self._do_exec()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1930, in _do_exec
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     self._execute_stmt(delete_stmt)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 1702, in _execute_stmt
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     self.result = self.query._execute_crud(stmt, self.mapper)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 3564, in _execute_crud
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     conn = self._connection_from_session(
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/query.py", line 3550, in _connection_from_session
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     conn = self.session.connection(**kw)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1142, in connection
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     return self._connection_for_bind(
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 1150, in _connection_for_bind
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     return self.transaction._connection_for_bind(
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     conn = bind._contextual_connect()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     self._wrap_pool_connect(self.pool.connect, None),
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     Connection._handle_dbapi_exception_noconnection(
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 1581, in _handle_dbapi_exception_noconnection
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     raise exception
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     return fn()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/base.py", line 364, in connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     return _ConnectionFairy._checkout(self)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     fairy = _ConnectionRecord.checkout(pool)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     rec = pool._do_get()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     self._dec_overflow()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     compat.raise_(
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     raise exception
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     return self._create_connection()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     return _ConnectionRecord(self)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     self.__connect(first_connect_check=True)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     pool.logger.debug("Error on connect(): %s", e)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     compat.raise_(
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     raise exception
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     connection = pool._invoke_creator(self)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     return dialect.connect(*cargs, **cparams)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.9/site-packages/sqlalchemy/engine/default.py", line 509, in connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     return self.dbapi.connect(*cargs, **cparams)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/pymysql/__init__.py", line 94, in Connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     return Connection(*args, **kwargs)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 327, in __init__
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     self.connect()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 587, in connect
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     self._get_server_information()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 969, in _get_server_information
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     packet = self._read_packet()
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 646, in _read_packet
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     packet_header = self._read_bytes(4)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.9/site-packages/pymysql/connections.py", line 698, in _read_bytes
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager     raise err.OperationalError(
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager (Background on this error at: http://sqlalche.me/e/13/e3q8)
2023-08-01 09:33:02.914 105 ERROR neutron_lib.callbacks.manager
2023-08-01 09:33:02.941 105 INFO neutron.wsgi [-] (105) wsgi starting up on http://[::1]:9696
2023-08-01 09:33:03.789 105 DEBUG neutron.wsgi [-] (105) accepted ('::1', 39026, 0, 0) server /usr/lib/python3.9/site-packages/eventlet/wsgi.py:992
2023-08-01 09:33:03.797 105 INFO neutron.wsgi [-] 172.17.1.147,::1 "GET /healthcheck HTTP/1.1" status: 200  len: 142 time: 0.0064757