1881783 – Cinder volume stuck in creating state after a DB Error

Bug 1881783 - Cinder volume stuck in creating state after a DB Error

Summary: Cinder volume stuck in creating state after a DB Error

Keywords:
Status:	CLOSED DUPLICATE of bug 1884263
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-cinder
Sub Component:
Version:	16.1 (Train)
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	Gorka Eguileor
QA Contact:	Tzach Shefi
Docs Contact:	Chuck Copello
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-09-23 05:13 UTC by PURANDHAR SAIRAM MANNIDI
Modified:	2023-12-15 19:32 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-01-28 05:36:33 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	OSP-1847	0	None	None	None	2023-12-15 19:32:13 UTC

Description PURANDHAR SAIRAM MANNIDI 2020-09-23 05:13:27 UTC

Description of problem:
Cinder volume stuck in creating / deleting state after a DB Error.

Version-Release number of selected component (if applicable):
Red Hat OpenStack Platform 16.1

How reproducible:
Always

Steps to Reproduce:
1. Launched 20 Heat Stacks with 20 second delay which creates 5 volumes / stack.
2. few volumes are stuck in creating state whenever a DB Error occurs.


Actual results:
few volumes are stuck in creating state whenever a DB Error occurs.

Expected results:
Volumes should either be retried for creation or catch the exception and move the volume into error state rather than creating / deleting.

Additional info:

Comment 1 Luigi Toscano 2020-09-23 07:37:57 UTC

Please note that the status is stored in the database, and if you can't reach the database you can't update the status, so this probably can't be fixed.

Comment 3 Pablo Caruana 2020-09-23 11:06:34 UTC

Talked with Sai, directly by irc.

His original observations were observed 'Create volume request issued successfully', then some  "2013, 'Lost connection to MySQL server during query'"
As ceph backend was showing "HEALTH_OK" pointed most the direction to the controller side.
From logs, "2020-09-21 11:39:14 1196201 [Warning] Aborted connection 1196201 to db: 'cinder' user: 'cinder' host: '10.170.214.20' (Got an error reading communication packets)" this raised the bell to check[1]:
 mysqladmin ext | grep Abort" kind for checking that stats (if something growing in time)
| Aborted_clients                                              | 6974                                                                                                                                                           
| Aborted_connects                                             | 296

As even nova was showing similar situation  a potential hint to corroborated was increasing  "connect_timeout in galera to 30 sec from 10", after that confirmed a lot of improvement, no errors till now according to [2] values tested were:

- increased mysql connect_timeout from 10 to 30 sec
- increased nova_libvirt block_device_allocate_retries to 200





@Sai could you confirm what is the current situation? is any pending action apart of what was discussed verbally?

[1]https://www.percona.com/blog/2008/08/23/how-to-track-down-the-source-of-aborted_connects/
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1872636#c32

Comment 4 PURANDHAR SAIRAM MANNIDI 2020-09-23 13:44:02 UTC

Things i changed:

MariaDB [(none)]> show variables like '%timeout%';
+---------------------------------------+----------+
| Variable_name                         | Value    |
+---------------------------------------+----------+
| connect_timeout                       | 120      | <<==== This one, Initially we raised to 30s which has improvement but didn't help and then 60s which had lot of improvement but still saw connect errors.
| deadlock_timeout_long                 | 50000000 |
| deadlock_timeout_short                | 10000    |
| delayed_insert_timeout                | 300      |
| idle_readonly_transaction_timeout     | 0        |
| idle_transaction_timeout              | 0        |
| idle_write_transaction_timeout        | 0        |
| innodb_flush_log_at_timeout           | 1        |
| innodb_lock_wait_timeout              | 120      |<< ==== This one for lock wait timeout.
| innodb_rollback_on_timeout            | OFF      |
| interactive_timeout                   | 28800    |
| lock_wait_timeout                     | 86400    |
| net_read_timeout                      | 30       |
| net_write_timeout                     | 60       |
| rpl_semi_sync_master_timeout          | 10000    |
| rpl_semi_sync_slave_kill_conn_timeout | 5        |
| slave_net_timeout                     | 60       |
| thread_pool_idle_timeout              | 60       |
| wait_timeout                          | 28800    |
+---------------------------------------+----------+


In Cinder.conf,

service_down_time = 300
report_interval = 30
rpc_response_timeout = 600
use_db_reconnect = True

In nova-compute:

block_device_allocate_retries=200

The above seem to help but there are still occasional mysql timeouts.

Comment 5 PURANDHAR SAIRAM MANNIDI 2020-09-23 13:56:59 UTC

Error I used to see earlier when connect_timeout was 10s / 30s:

        |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7f885eb29be0>, 'volume': Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=True,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2020-09-23T06:14:20Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=8b378180-9424-4378-9dc5-cb9b79faf85c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='af870d09307d4601b37da4a510d4821e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2020-09-23T06:14:20Z,service_uuid=None,shared_targets=True,size=30,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2020-09-23T06:14:20Z,user_id='4ff79464480c468ea35e60dee965b82b',volume_attachment=<?>,volume_type=VolumeType(6921b4ce-23d7-489a-884a-e56ed8f4e366),volume_type_id=6921b4ce-23d7-489a-884a-e56ed8f4e366)}, 'provides': Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=True,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2020-09-23T06:14:20Z,deleted=False,deleted_at=None,display_description='',display_name='',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='hostgroup@tripleo_ceph#tripleo_ceph',id=8b378180-9424-4378-9dc5-cb9b79faf85c,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='af870d09307d4601b37da4a510d4821e',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2020-09-23T06:14:20Z,service_uuid=None,shared_targets=True,size=30,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2020-09-23T06:14:20Z,user_id='4ff79464480c468ea35e60dee965b82b',volume_attachment=<?>,volume_type=VolumeType(6921b4ce-23d7-489a-884a-e56ed8f4e366),volume_type_id=6921b4ce-23d7-489a-884a-e56ed8f4e366)}
           |__Flow 'volume_create_manager': oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
(Background on this error at: http://sqlalche.me/e/e3q8)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager Traceback (most recent call last):
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return fn()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 354, in connect
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return _ConnectionFairy._checkout(self)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 751, in _checkout
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     fairy = _ConnectionRecord.checkout(pool)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 483, in checkout
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     rec = pool._do_get()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 138, in _do_get
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     self._dec_overflow()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     compat.reraise(exc_type, exc_value, exc_tb)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 129, in reraise
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     raise value
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 135, in _do_get
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return self._create_connection()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 299, in _create_connection
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return _ConnectionRecord(self)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 428, in __init__
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     self.__connect(first_connect_check=True)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 630, in __connect
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     connection = pool._invoke_creator(self)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return dialect.connect(*cargs, **cparams)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 453, in connect
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return self.dbapi.connect(*cargs, **cparams)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/pymysql/__init__.py", line 90, in Connect
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return Connection(*args, **kwargs)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 699, in __init__
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     self.connect()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 936, in connect
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     self._request_authentication()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1156, in _request_authentication
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     auth_packet = self._read_packet()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 991, in _read_packet
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     packet_header = self._read_bytes(4)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1037, in _read_bytes
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager The above exception was the direct cause of the following exception:
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager Traceback (most recent call last):
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     result = task.execute(**arguments)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1135, in execute
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     **volume_spec)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 696, in _wrapper
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return r.call(f, *args, **kwargs)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return attempt.get(self._wrap_exception)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     six.reraise(self.value[0], self.value[1], self.value[2])
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     raise value
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1039, in _create_from_image
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     image_meta=image_meta)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 412, in _handle_bootable_volume_glance_meta
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     volume_utils.enable_bootable_flag(volume)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/volume/volume_utils.py", line 1091, in enable_bootable_flag
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     volume.save()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/objects/volume.py", line 399, in save
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     db.volume_update(self._context, self.id, updates)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/db/api.py", line 326, in volume_update
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return IMPL.volume_update(context, volume_id, values)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 241, in wrapper
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return f(*args, **kwargs)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return f(*args, **kwargs)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 2590, in volume_update
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     result = query.filter_by(id=volume_id).update(values)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3818, in update
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     update_op.exec_()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1671, in exec_
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     self._do_exec()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1864, in _do_exec
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     self._execute_stmt(update_stmt)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1676, in _execute_stmt
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     self.result = self.query._execute_crud(stmt, self.mapper)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3347, in _execute_crud
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     mapper=mapper, clause=stmt, close_with_result=True
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3332, in _connection_from_session
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     conn = self.session.connection(**kw)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1123, in connection
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     execution_options=execution_options,
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1129, in _connection_for_bind
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     engine, execution_options
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 430, in _connection_for_bind
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     conn = bind._contextual_connect()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     self._wrap_pool_connect(self.pool.connect, None),
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     e, dialect, self
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1534, in _handle_dbapi_exception_noconnection
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     util.raise_from_cause(newraise, exc_info)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     raise value.with_traceback(tb)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return fn()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 354, in connect
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return _ConnectionFairy._checkout(self)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 751, in _checkout
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     fairy = _ConnectionRecord.checkout(pool)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 483, in checkout
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     rec = pool._do_get()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 138, in _do_get
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     self._dec_overflow()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     compat.reraise(exc_type, exc_value, exc_tb)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 129, in reraise
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     raise value
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 135, in _do_get
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return self._create_connection()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 299, in _create_connection
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return _ConnectionRecord(self)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 428, in __init__
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     self.__connect(first_connect_check=True)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 630, in __connect
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     connection = pool._invoke_creator(self)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return dialect.connect(*cargs, **cparams)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 453, in connect
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return self.dbapi.connect(*cargs, **cparams)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/pymysql/__init__.py", line 90, in Connect
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     return Connection(*args, **kwargs)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 699, in __init__
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     self.connect()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 936, in connect
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     self._request_authentication()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1156, in _request_authentication
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     auth_packet = self._read_packet()
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 991, in _read_packet
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     packet_header = self._read_bytes(4)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 1037, in _read_bytes
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager     CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager (Background on this error at: http://sqlalche.me/e/e3q8)
2020-09-23 06:14:39.051 47 ERROR cinder.volume.manager

Comment 6 PURANDHAR SAIRAM MANNIDI 2020-09-23 13:59:23 UTC

After setting connect_timeout in galera to 120s, I see the following errors:

2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server [req-7db01864-d085-42e4-b36a-7ddcb5800e5e 4ff79464480c468ea35e60dee965b82b af870d09307d4601b37da4a510d4821e - default default] Exception during message handling: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.170.214.12' (timed out)")
(Background on this error at: http://sqlalche.me/e/e3q8)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 920, in connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     **kwargs)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/green/socket.py", line 63, in create_connection
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     raise err
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/green/socket.py", line 53, in create_connection
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     sock.connect(sa)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 263, in connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     self._trampoline(fd, write=True, timeout=timeout, timeout_exc=_timeout_exc)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 210, in _trampoline
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     mark_as_closed=self._mark_as_closed)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/hubs/__init__.py", line 159, in trampoline
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return hub.switch()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return self.greenlet.switch()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server socket.timeout: timed out
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return fn()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 354, in connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return _ConnectionFairy._checkout(self)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 751, in _checkout
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     fairy = _ConnectionRecord.checkout(pool)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 483, in checkout
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     rec = pool._do_get()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 138, in _do_get
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     self._dec_overflow()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     compat.reraise(exc_type, exc_value, exc_tb)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 129, in reraise
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     raise value
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 135, in _do_get
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return self._create_connection()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 299, in _create_connection
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return _ConnectionRecord(self)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 428, in __init__
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     self.__connect(first_connect_check=True)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 630, in __connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     connection = pool._invoke_creator(self)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return dialect.connect(*cargs, **cparams)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 453, in connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return self.dbapi.connect(*cargs, **cparams)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/pymysql/__init__.py", line 90, in Connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return Connection(*args, **kwargs)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 699, in __init__
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     self.connect()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 967, in connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     raise exc
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on '10.170.214.12' (timed out)")
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server The above exception was the direct cause of the following exception:
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "<decorator-gen-233>", line 2, in create_volume
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/objects/cleanable.py", line 212, in wrapper
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/manager.py", line 749, in create_volume
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     _run_flow()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/manager.py", line 741, in _run_flow
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     flow_engine.run()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     for _state in self.run_iter(timeout=timeout):
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     failure.Failure.reraise_if_any(er_failures)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/taskflow/types/failure.py", line 339, in reraise_if_any
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     failures[0].reraise()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/taskflow/types/failure.py", line 346, in reraise
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     six.reraise(*self._exc_info)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     raise value
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     result = task.execute(**arguments)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 239, in execute
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     volume.refresh()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/objects/cleanable.py", line 242, in refresh
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     super(CinderCleanableObject, self).refresh()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/objects/base.py", line 492, in refresh
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     current = self.get_by_id(self._context, self.id)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/objects/base.py", line 352, in get_by_id
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/db/api.py", line 1801, in get_by_id
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return IMPL.get_by_id(context, model, id, *args, **kwargs)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 7123, in get_by_id
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return _GET_METHODS[model](context, id, *args, **kwargs)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 2137, in volume_get
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return _volume_get(context, volume_id)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 1897, in _volume_get
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     result = result.filter_by(id=volume_id).first()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3215, in first
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     ret = list(self[0:1])
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3007, in __getitem__
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return list(res)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3317, in __iter__
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return self._execute_and_instances(context)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3339, in _execute_and_instances
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     querycontext, self._connection_from_session, close_with_result=True
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3354, in _get_bind_args
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     mapper=self._bind_mapper(), clause=querycontext.statement, **kw
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3332, in _connection_from_session
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     conn = self.session.connection(**kw)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1123, in connection
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     execution_options=execution_options,
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1132, in _connection_for_bind
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     conn = engine._contextual_connect(**kw)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     self._wrap_pool_connect(self.pool.connect, None),
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     e, dialect, self
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1534, in _handle_dbapi_exception_noconnection
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     util.raise_from_cause(newraise, exc_info)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     raise value.with_traceback(tb)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return fn()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 354, in connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return _ConnectionFairy._checkout(self)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 751, in _checkout
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     fairy = _ConnectionRecord.checkout(pool)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 483, in checkout
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     rec = pool._do_get()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 138, in _do_get
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     self._dec_overflow()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     compat.reraise(exc_type, exc_value, exc_tb)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 129, in reraise
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     raise value
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 135, in _do_get
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return self._create_connection()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 299, in _create_connection
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return _ConnectionRecord(self)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 428, in __init__
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     self.__connect(first_connect_check=True)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 630, in __connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     connection = pool._invoke_creator(self)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return dialect.connect(*cargs, **cparams)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 453, in connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return self.dbapi.connect(*cargs, **cparams)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/pymysql/__init__.py", line 90, in Connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     return Connection(*args, **kwargs)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 699, in __init__
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     self.connect()
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 967, in connect
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server     raise exc
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.170.214.12' (timed out)")
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server (Background on this error at: http://sqlalche.me/e/e3q8)
2020-09-23 13:09:43.367 47 ERROR oslo_messaging.rpc.server

Comment 13 PURANDHAR SAIRAM MANNIDI 2021-01-28 05:36:33 UTC


*** This bug has been marked as a duplicate of bug 1884263 ***

Note You need to log in before you can comment on or make changes to this bug.