Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1888011 - When flattening RBD images, cinder-volume losses connectivity to rabbitmq due to client heartbeat timeout [NEEDINFO]
Summary: When flattening RBD images, cinder-volume losses connectivity to rabbitmq due...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 16.1 (Train)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: z3
: 16.1 (Train on RHEL 8.2)
Assignee: Eric Harney
QA Contact: Tzach Shefi
Chuck Copello
URL:
Whiteboard:
: 1894303 (view as bug list)
Depends On:
Blocks: 1885734 1894303
TreeView+ depends on / blocked
 
Reported: 2020-10-13 19:44 UTC by Eric Harney
Modified: 2021-01-07 15:01 UTC (History)
7 users (show)

Fixed In Version: openstack-cinder-15.3.1-1.20200915213430.8181c4b.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-12-15 18:37:11 UTC
Target Upstream Version:
chrisbro: needinfo? (eharney)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2020:5413 0 None None None 2020-12-15 18:37:38 UTC

Internal Links: 1931204

Description Eric Harney 2020-10-13 19:44:17 UTC
This bug was initially created as a copy of Bug #1885734


Description of problem:

When flattening RBD images, cinder-volume losses connectivity to rabbitmq due to client heartbeat timeout and subsequent volume creation appears to be failing:


2020-10-06 15:15:50.746 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:15:50.846 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] creating snapshot='volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72.clone_snap' create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:588
2020-10-06 15:15:51.585 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] cloning 'volume-d5b6f370-11b9-4561-b8e6-ec40bf049cca@volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72.clone_snap' to 'volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72' create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:597
2020-10-06 15:15:52.003 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] Volume volume-eb0c640e-f22e-4b49-8675-cff8bc39ea26.deleted is not a clone. _get_clone_info /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:972
2020-10-06 15:15:52.005 55 INFO cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] maximum clone depth (5) has been reached - flattening dest volume
2020-10-06 15:15:52.091 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] flattening dest volume volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72 create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:625
2020-10-06 15:15:55.586 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:15:55.619 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:15:55.656 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:16:41.979 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:16:42.017 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:16:42.052 110 DEBUG cinder.manager [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:16:55.591 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:16:55.623 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:16:55.661 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:17:41.976 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:17:42.012 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:17:42.047 110 DEBUG cinder.manager [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:17:55.586 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:17:55.623 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:17:55.666 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:18:41.981 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:18:42.016 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:18:42.049 110 DEBUG cinder.manager [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:18:55.591 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:18:55.624 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:18:55.657 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:19:05.575 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] remove temporary snap volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72.clone_snap create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:640
2020-10-06 15:19:05.688 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] clone created successfully create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:666
2020-10-06 15:19:05.690 55 DEBUG cinder.volume.manager [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5b07b48e-e622-4603-b45b-81ac5099ea67) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': u'creating', 'volume_size': 200, 'volume_name': u'volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72', 'source_volid': u'd5b6f370-11b9-4561-b8e6-ec40bf049cca', 'source_volstatus': u'in-use'}' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183
2020-10-06 15:19:05.691 55 DEBUG cinder.volume.manager [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (3addaf14-ccd7-4889-8dc0-1b2b4d880911) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194
2020-10-06 15:19:05.694 55 DEBUG oslo.messaging._drivers.impl_rabbit [-] [a2e721c1-7e9d-42d4-af9c-df2f9c2c7322] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:765
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, **kwargs)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, channel=channels[0], **kwargs), channels[0]
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 816, in execute_method
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     method()
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1103, in _consume
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     self._heartbeat_check()
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 969, in _heartbeat_check
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     self.connection.heartbeat_check(rate=self.heartbeat_rate)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 290, in heartbeat_check
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return self.transport.heartbeat_check(self.connection, rate=rate)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 149, in heartbeat_check
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return connection.heartbeat_tick(rate=rate)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 718, in heartbeat_tick
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     self.send_heartbeat()
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 669, in send_heartbeat
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     self.frame_writer(8, 0, None, None, None)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 166, in write_frame
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     write(view[:offset])
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 275, in write
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     self._write(s)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 390, in sendall
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     tail = self.send(data, flags)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in send
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return self._send_loop(self.fd.send, data, flags)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in _send_loop
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return send_method(data, *args)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit 
2020-10-06 15:19:05.696 55 DEBUG oslo.messaging._drivers.impl_rabbit [-] [a2e721c1-7e9d-42d4-af9c-df2f9c2c7322] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:765
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, **kwargs)
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, channel=channels[0], **kwargs), channels[0]
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 816, in execute_method
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit     method()
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1088, in _consume
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit     raise self.connection.recoverable_connection_errors[0]
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit RecoverableConnectionError: <AMQPError: unknown error>
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit 
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2e721c1-7e9d-42d4-af9c-df2f9c2c7322] AMQP server on overcloud-controller-1:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
2020-10-06 15:19:05.697 55 DEBUG oslo.messaging._drivers.impl_rabbit [-] [80610839-4914-4df7-b1ab-d5d0efcc2fae] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:765
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, **kwargs)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, channel=channels[0], **kwargs), channels[0]
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 816, in execute_method
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     method()
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1103, in _consume
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     self._heartbeat_check()
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 969, in _heartbeat_check
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     self.connection.heartbeat_check(rate=self.heartbeat_rate)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 290, in heartbeat_check
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return self.transport.heartbeat_check(self.connection, rate=rate)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 149, in heartbeat_check
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return connection.heartbeat_tick(rate=rate)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 718, in heartbeat_tick
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     self.send_heartbeat()
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 669, in send_heartbeat
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     self.frame_writer(8, 0, None, None, None)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 166, in write_frame
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     write(view[:offset])
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 275, in write
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     self._write(s)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 390, in sendall
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     tail = self.send(data, flags)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in send
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return self._send_loop(self.fd.send, data, flags)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in _send_loop
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return send_method(data, *args)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit 
2020-10-06 15:19:05.698 55 ERROR oslo.messaging._drivers.impl_rabbit [-] [80610839-4914-4df7-b1ab-d5d0efcc2fae] AMQP server on overcloud-controller-0:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer



rabbitmq logs:
~~~
These are in the rabbitmq logs

=ERROR REPORT==== 6-Oct-2020::16:55:41 ===
closing AMQP connection <0.20256.0> (10.10.10.10:39952 -> 10.10.10.11:5672 - cinder-volume:55:b8e9ab46-ab18-45e6-835d-78aa1756bd2e):
missed heartbeats from client, timeout: 60s
~~~

Comment 6 Takashi Kajinami 2020-11-03 23:41:45 UTC
*** Bug 1894303 has been marked as a duplicate of this bug. ***

Comment 17 Tzach Shefi 2020-11-18 19:33:09 UTC
Verified on:
openstack-cinder-15.3.1-5.el8ost.noarch


Created a Cinder volume volA, filled it with some data. 
(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --source-volid f135f778-fab3-44d1-901d-22c0e1330919  --name volB+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2020-11-18T19:24:13.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 015b6c15-6134-4e3b-a747-157fc7c85ec4 |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | volB                                 |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 23cfee0f63b44e31a44761d5da636209     |
| replication_status             | None                                 |
| size                           | 1                                    |
| snapshot_id                    | None                                 |
| source_volid                   | f135f778-fab3-44d1-901d-22c0e1330919 |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | cdb4cc954bc14b1db71b5b14fe87fa2c     |
| volume_type                    | tripleo                              |
+--------------------------------+--------------------------------------+

(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --source-volid 015b6c15-6134-4e3b-a747-157fc7c85ec4  --name volC
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2020-11-18T19:25:08.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8 |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | volC                                 |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 23cfee0f63b44e31a44761d5da636209     |
| replication_status             | None                                 |
| size                           | 1                                    |
| snapshot_id                    | None                                 |
| source_volid                   | 015b6c15-6134-4e3b-a747-157fc7c85ec4 |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | cdb4cc954bc14b1db71b5b14fe87fa2c     |
| volume_type                    | tripleo                              |
+--------------------------------+--------------------------------------+


(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --source-volid 2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8  --name volD
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2020-11-18T19:25:34.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | volD                                 |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 23cfee0f63b44e31a44761d5da636209     |
| replication_status             | None                                 |
| size                           | 1                                    |
| snapshot_id                    | None                                 |
| source_volid                   | 2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8 |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | cdb4cc954bc14b1db71b5b14fe87fa2c     |
| volume_type                    | tripleo                              |
+--------------------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --source-volid bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f  --name volE
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2020-11-18T19:25:58.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 73a45018-6664-491a-b754-9c59be267807 |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | volE                                 |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 23cfee0f63b44e31a44761d5da636209     |
| replication_status             | None                                 |
| size                           | 1                                    |
| snapshot_id                    | None                                 |
| source_volid                   | bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | cdb4cc954bc14b1db71b5b14fe87fa2c     |
| volume_type                    | tripleo                              |
+--------------------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --source-volid 73a45018-6664-491a-b754-9c59be267807  --name volF
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2020-11-18T19:26:16.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 3f9356de-e3ab-4d0b-a2c9-7d02bc26a68c |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | volF                                 |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 23cfee0f63b44e31a44761d5da636209     |
| replication_status             | None                                 |
| size                           | 1                                    |
| snapshot_id                    | None                                 |
| source_volid                   | 73a45018-6664-491a-b754-9c59be267807 |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | cdb4cc954bc14b1db71b5b14fe87fa2c     |
| volume_type                    | tripleo                              |
+--------------------------------+--------------------------------------+


Once we passed the depth of 5 clones with volF,we noticed the flatten operation on c-vol.
Also per bz verification no errors or tracebacks show up now, which did happen before the fix.  

C-vol log reports:

2020-11-18 19:22:33.934 7 DEBUG oslo_service.service [req-d64dbea1-34f3-4e6e-a319-ad94e6c21036 - - - - -] backend_defaults.rbd_max_clone_depth = 5 log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2589
[root@controller-2 ~]# grep depth /var/log/containers/cinder/cinder-volume.log
2020-11-18 18:18:56.561 39 INFO cinder.volume.drivers.rbd [req-abc397a1-4c67-4c56-be7f-6456b204876d cdb4cc954bc14b1db71b5b14fe87fa2c 23cfee0f63b44e31a44761d5da636209 - default default] maximum clone depth (1) has been reached - flattening dest volume
2020-11-18 18:20:04.989 39 INFO cinder.volume.drivers.rbd [req-b5fe1c63-69d0-421a-ac6c-068e065bedfb cdb4cc954bc14b1db71b5b14fe87fa2c 23cfee0f63b44e31a44761d5da636209 - default default] maximum clone depth (1) has been reached - flattening dest volume

Flatting works, we have no errors or traceback on c-vol log. 

Ceph backend:

[root@controller-2 /]#  rbd -p volumes ls -l
NAME                                                                                               SIZE  PARENT                                                                                                     FMT PROT LOCK 
volume-015b6c15-6134-4e3b-a747-157fc7c85ec4                                                        1 GiB volumes/volume-f135f778-fab3-44d1-901d-22c0e1330919@volume-015b6c15-6134-4e3b-a747-157fc7c85ec4.clone_snap   2           
volume-015b6c15-6134-4e3b-a747-157fc7c85ec4@volume-2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8.clone_snap 1 GiB volumes/volume-f135f778-fab3-44d1-901d-22c0e1330919@volume-015b6c15-6134-4e3b-a747-157fc7c85ec4.clone_snap   2 yes       
volume-0a6b6446-4717-453b-b7f6-c92155665a28.deleted                                                1 GiB                                                                                                              2           
volume-2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8                                                        1 GiB volumes/volume-015b6c15-6134-4e3b-a747-157fc7c85ec4@volume-2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8.clone_snap   2           
volume-2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8@volume-bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f.clone_snap 1 GiB volumes/volume-015b6c15-6134-4e3b-a747-157fc7c85ec4@volume-2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8.clone_snap   2 yes       
volume-3f9356de-e3ab-4d0b-a2c9-7d02bc26a68c                                                        1 GiB volumes/volume-73a45018-6664-491a-b754-9c59be267807@volume-3f9356de-e3ab-4d0b-a2c9-7d02bc26a68c.clone_snap   2           
volume-73a45018-6664-491a-b754-9c59be267807                                                        1 GiB volumes/volume-bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f@volume-73a45018-6664-491a-b754-9c59be267807.clone_snap   2           
volume-73a45018-6664-491a-b754-9c59be267807@volume-3f9356de-e3ab-4d0b-a2c9-7d02bc26a68c.clone_snap 1 GiB volumes/volume-bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f@volume-73a45018-6664-491a-b754-9c59be267807.clone_snap   2 yes       
volume-bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f                                                        1 GiB volumes/volume-2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8@volume-bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f.clone_snap   2           
volume-bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f@volume-73a45018-6664-491a-b754-9c59be267807.clone_snap 1 GiB volumes/volume-2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8@volume-bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f.clone_snap   2 yes       
volume-f135f778-fab3-44d1-901d-22c0e1330919                                                        1 GiB                                                                                                              2           
volume-f135f778-fab3-44d1-901d-22c0e1330919@volume-015b6c15-6134-4e3b-a747-157fc7c85ec4.clone_snap 1 GiB                                                                                                              2 yes

Comment 24 errata-xmlrpc 2020-12-15 18:37:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.3 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:5413


Note You need to log in before you can comment on or make changes to this bug.