This bug was initially created as a copy of Bug #1885734 Description of problem: When flattening RBD images, cinder-volume losses connectivity to rabbitmq due to client heartbeat timeout and subsequent volume creation appears to be failing: 2020-10-06 15:15:50.746 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:15:50.846 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] creating snapshot='volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72.clone_snap' create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:588 2020-10-06 15:15:51.585 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] cloning 'volume-d5b6f370-11b9-4561-b8e6-ec40bf049cca' to 'volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72' create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:597 2020-10-06 15:15:52.003 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] Volume volume-eb0c640e-f22e-4b49-8675-cff8bc39ea26.deleted is not a clone. _get_clone_info /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:972 2020-10-06 15:15:52.005 55 INFO cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] maximum clone depth (5) has been reached - flattening dest volume 2020-10-06 15:15:52.091 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] flattening dest volume volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72 create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:625 2020-10-06 15:15:55.586 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:15:55.619 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:15:55.656 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194 2020-10-06 15:16:41.979 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:16:42.017 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:16:42.052 110 DEBUG cinder.manager [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194 2020-10-06 15:16:55.591 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:16:55.623 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:16:55.661 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194 2020-10-06 15:17:41.976 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:17:42.012 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:17:42.047 110 DEBUG cinder.manager [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194 2020-10-06 15:17:55.586 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:17:55.623 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:17:55.666 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194 2020-10-06 15:18:41.981 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:18:42.016 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:18:42.049 110 DEBUG cinder.manager [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194 2020-10-06 15:18:55.591 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:18:55.624 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:18:55.657 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194 2020-10-06 15:19:05.575 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] remove temporary snap volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72.clone_snap create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:640 2020-10-06 15:19:05.688 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] clone created successfully create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:666 2020-10-06 15:19:05.690 55 DEBUG cinder.volume.manager [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5b07b48e-e622-4603-b45b-81ac5099ea67) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': u'creating', 'volume_size': 200, 'volume_name': u'volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72', 'source_volid': u'd5b6f370-11b9-4561-b8e6-ec40bf049cca', 'source_volstatus': u'in-use'}' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 2020-10-06 15:19:05.691 55 DEBUG cinder.volume.manager [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (3addaf14-ccd7-4889-8dc0-1b2b4d880911) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 2020-10-06 15:19:05.694 55 DEBUG oslo.messaging._drivers.impl_rabbit [-] [a2e721c1-7e9d-42d4-af9c-df2f9c2c7322] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:765 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__ 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0] 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 816, in execute_method 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit method() 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1103, in _consume 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit self._heartbeat_check() 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 969, in _heartbeat_check 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit self.connection.heartbeat_check(rate=self.heartbeat_rate) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 290, in heartbeat_check 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit return self.transport.heartbeat_check(self.connection, rate=rate) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 149, in heartbeat_check 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit return connection.heartbeat_tick(rate=rate) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 718, in heartbeat_tick 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit self.send_heartbeat() 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 669, in send_heartbeat 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit self.frame_writer(8, 0, None, None, None) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 166, in write_frame 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit write(view[:offset]) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 275, in write 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit self._write(s) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 390, in sendall 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in send 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit return self._send_loop(self.fd.send, data, flags) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in _send_loop 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit return send_method(data, *args) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit 2020-10-06 15:19:05.696 55 DEBUG oslo.messaging._drivers.impl_rabbit [-] [a2e721c1-7e9d-42d4-af9c-df2f9c2c7322] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:765 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs) 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__ 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0] 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 816, in execute_method 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit method() 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1088, in _consume 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit raise self.connection.recoverable_connection_errors[0] 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit RecoverableConnectionError: <AMQPError: unknown error> 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2e721c1-7e9d-42d4-af9c-df2f9c2c7322] AMQP server on overcloud-controller-1:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error> 2020-10-06 15:19:05.697 55 DEBUG oslo.messaging._drivers.impl_rabbit [-] [80610839-4914-4df7-b1ab-d5d0efcc2fae] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:765 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__ 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0] 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 816, in execute_method 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit method() 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1103, in _consume 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit self._heartbeat_check() 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 969, in _heartbeat_check 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit self.connection.heartbeat_check(rate=self.heartbeat_rate) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 290, in heartbeat_check 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit return self.transport.heartbeat_check(self.connection, rate=rate) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 149, in heartbeat_check 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit return connection.heartbeat_tick(rate=rate) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 718, in heartbeat_tick 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit self.send_heartbeat() 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 669, in send_heartbeat 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit self.frame_writer(8, 0, None, None, None) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 166, in write_frame 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit write(view[:offset]) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 275, in write 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit self._write(s) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 390, in sendall 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in send 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit return self._send_loop(self.fd.send, data, flags) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in _send_loop 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit return send_method(data, *args) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit 2020-10-06 15:19:05.698 55 ERROR oslo.messaging._drivers.impl_rabbit [-] [80610839-4914-4df7-b1ab-d5d0efcc2fae] AMQP server on overcloud-controller-0:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer rabbitmq logs: ~~~ These are in the rabbitmq logs =ERROR REPORT==== 6-Oct-2020::16:55:41 === closing AMQP connection <0.20256.0> (10.10.10.10:39952 -> 10.10.10.11:5672 - cinder-volume:55:b8e9ab46-ab18-45e6-835d-78aa1756bd2e): missed heartbeats from client, timeout: 60s ~~~
*** Bug 1894303 has been marked as a duplicate of this bug. ***
Verified on: openstack-cinder-15.3.1-5.el8ost.noarch Created a Cinder volume volA, filled it with some data. (overcloud) [stack@undercloud-0 ~]$ cinder create 1 --source-volid f135f778-fab3-44d1-901d-22c0e1330919 --name volB+--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2020-11-18T19:24:13.000000 | | description | None | | encrypted | False | | id | 015b6c15-6134-4e3b-a747-157fc7c85ec4 | | metadata | {} | | migration_status | None | | multiattach | False | | name | volB | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 23cfee0f63b44e31a44761d5da636209 | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | f135f778-fab3-44d1-901d-22c0e1330919 | | status | creating | | updated_at | None | | user_id | cdb4cc954bc14b1db71b5b14fe87fa2c | | volume_type | tripleo | +--------------------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ cinder create 1 --source-volid 015b6c15-6134-4e3b-a747-157fc7c85ec4 --name volC +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2020-11-18T19:25:08.000000 | | description | None | | encrypted | False | | id | 2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8 | | metadata | {} | | migration_status | None | | multiattach | False | | name | volC | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 23cfee0f63b44e31a44761d5da636209 | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | 015b6c15-6134-4e3b-a747-157fc7c85ec4 | | status | creating | | updated_at | None | | user_id | cdb4cc954bc14b1db71b5b14fe87fa2c | | volume_type | tripleo | +--------------------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ cinder create 1 --source-volid 2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8 --name volD +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2020-11-18T19:25:34.000000 | | description | None | | encrypted | False | | id | bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f | | metadata | {} | | migration_status | None | | multiattach | False | | name | volD | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 23cfee0f63b44e31a44761d5da636209 | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | 2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8 | | status | creating | | updated_at | None | | user_id | cdb4cc954bc14b1db71b5b14fe87fa2c | | volume_type | tripleo | +--------------------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ cinder create 1 --source-volid bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f --name volE +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2020-11-18T19:25:58.000000 | | description | None | | encrypted | False | | id | 73a45018-6664-491a-b754-9c59be267807 | | metadata | {} | | migration_status | None | | multiattach | False | | name | volE | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 23cfee0f63b44e31a44761d5da636209 | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f | | status | creating | | updated_at | None | | user_id | cdb4cc954bc14b1db71b5b14fe87fa2c | | volume_type | tripleo | +--------------------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ cinder create 1 --source-volid 73a45018-6664-491a-b754-9c59be267807 --name volF +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2020-11-18T19:26:16.000000 | | description | None | | encrypted | False | | id | 3f9356de-e3ab-4d0b-a2c9-7d02bc26a68c | | metadata | {} | | migration_status | None | | multiattach | False | | name | volF | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 23cfee0f63b44e31a44761d5da636209 | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | 73a45018-6664-491a-b754-9c59be267807 | | status | creating | | updated_at | None | | user_id | cdb4cc954bc14b1db71b5b14fe87fa2c | | volume_type | tripleo | +--------------------------------+--------------------------------------+ Once we passed the depth of 5 clones with volF,we noticed the flatten operation on c-vol. Also per bz verification no errors or tracebacks show up now, which did happen before the fix. C-vol log reports: 2020-11-18 19:22:33.934 7 DEBUG oslo_service.service [req-d64dbea1-34f3-4e6e-a319-ad94e6c21036 - - - - -] backend_defaults.rbd_max_clone_depth = 5 log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2589 [root@controller-2 ~]# grep depth /var/log/containers/cinder/cinder-volume.log 2020-11-18 18:18:56.561 39 INFO cinder.volume.drivers.rbd [req-abc397a1-4c67-4c56-be7f-6456b204876d cdb4cc954bc14b1db71b5b14fe87fa2c 23cfee0f63b44e31a44761d5da636209 - default default] maximum clone depth (1) has been reached - flattening dest volume 2020-11-18 18:20:04.989 39 INFO cinder.volume.drivers.rbd [req-b5fe1c63-69d0-421a-ac6c-068e065bedfb cdb4cc954bc14b1db71b5b14fe87fa2c 23cfee0f63b44e31a44761d5da636209 - default default] maximum clone depth (1) has been reached - flattening dest volume Flatting works, we have no errors or traceback on c-vol log. Ceph backend: [root@controller-2 /]# rbd -p volumes ls -l NAME SIZE PARENT FMT PROT LOCK volume-015b6c15-6134-4e3b-a747-157fc7c85ec4 1 GiB volumes/volume-f135f778-fab3-44d1-901d-22c0e1330919 2 volume-015b6c15-6134-4e3b-a747-157fc7c85ec4 1 GiB volumes/volume-f135f778-fab3-44d1-901d-22c0e1330919 2 yes volume-0a6b6446-4717-453b-b7f6-c92155665a28.deleted 1 GiB 2 volume-2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8 1 GiB volumes/volume-015b6c15-6134-4e3b-a747-157fc7c85ec4 2 volume-2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8 1 GiB volumes/volume-015b6c15-6134-4e3b-a747-157fc7c85ec4 2 yes volume-3f9356de-e3ab-4d0b-a2c9-7d02bc26a68c 1 GiB volumes/volume-73a45018-6664-491a-b754-9c59be267807 2 volume-73a45018-6664-491a-b754-9c59be267807 1 GiB volumes/volume-bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f 2 volume-73a45018-6664-491a-b754-9c59be267807 1 GiB volumes/volume-bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f 2 yes volume-bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f 1 GiB volumes/volume-2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8 2 volume-bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f 1 GiB volumes/volume-2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8 2 yes volume-f135f778-fab3-44d1-901d-22c0e1330919 1 GiB 2 volume-f135f778-fab3-44d1-901d-22c0e1330919 1 GiB 2 yes
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.3 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:5413
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days