Bug 1888011

Summary:	When flattening RBD images, cinder-volume losses connectivity to rabbitmq due to client heartbeat timeout
Product:	Red Hat OpenStack	Reporter:	Eric Harney <eharney>
Component:	openstack-cinder	Assignee:	Eric Harney <eharney>
Status:	CLOSED ERRATA	QA Contact:	Tzach Shefi <tshefi>
Severity:	medium	Docs Contact:	Chuck Copello <ccopello>
Priority:	medium
Version:	16.1 (Train)	CC:	abishop, chrisbro, dhill, gcharot, irichart, senrique, tkajinam
Target Milestone:	z3	Keywords:	Triaged, ZStream
Target Release:	16.1 (Train on RHEL 8.2)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-cinder-15.3.1-1.20200915213430.8181c4b.el8ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-12-15 18:37:11 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1885734, 1894303

Description Eric Harney 2020-10-13 19:44:17 UTC

This bug was initially created as a copy of Bug #1885734


Description of problem:

When flattening RBD images, cinder-volume losses connectivity to rabbitmq due to client heartbeat timeout and subsequent volume creation appears to be failing:


2020-10-06 15:15:50.746 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:15:50.846 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] creating snapshot='volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72.clone_snap' create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:588
2020-10-06 15:15:51.585 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] cloning 'volume-d5b6f370-11b9-4561-b8e6-ec40bf049cca' to 'volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72' create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:597
2020-10-06 15:15:52.003 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] Volume volume-eb0c640e-f22e-4b49-8675-cff8bc39ea26.deleted is not a clone. _get_clone_info /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:972
2020-10-06 15:15:52.005 55 INFO cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] maximum clone depth (5) has been reached - flattening dest volume
2020-10-06 15:15:52.091 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] flattening dest volume volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72 create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:625
2020-10-06 15:15:55.586 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:15:55.619 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:15:55.656 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:16:41.979 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:16:42.017 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:16:42.052 110 DEBUG cinder.manager [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:16:55.591 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:16:55.623 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:16:55.661 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:17:41.976 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:17:42.012 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:17:42.047 110 DEBUG cinder.manager [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:17:55.586 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:17:55.623 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:17:55.666 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:18:41.981 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:18:42.016 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:18:42.049 110 DEBUG cinder.manager [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:18:55.591 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:18:55.624 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337
2020-10-06 15:18:55.657 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194
2020-10-06 15:19:05.575 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] remove temporary snap volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72.clone_snap create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:640
2020-10-06 15:19:05.688 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] clone created successfully create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:666
2020-10-06 15:19:05.690 55 DEBUG cinder.volume.manager [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5b07b48e-e622-4603-b45b-81ac5099ea67) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': u'creating', 'volume_size': 200, 'volume_name': u'volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72', 'source_volid': u'd5b6f370-11b9-4561-b8e6-ec40bf049cca', 'source_volstatus': u'in-use'}' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183
2020-10-06 15:19:05.691 55 DEBUG cinder.volume.manager [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (3addaf14-ccd7-4889-8dc0-1b2b4d880911) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194
2020-10-06 15:19:05.694 55 DEBUG oslo.messaging._drivers.impl_rabbit [-] [a2e721c1-7e9d-42d4-af9c-df2f9c2c7322] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:765
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, **kwargs)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, channel=channels[0], **kwargs), channels[0]
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 816, in execute_method
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     method()
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1103, in _consume
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     self._heartbeat_check()
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 969, in _heartbeat_check
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     self.connection.heartbeat_check(rate=self.heartbeat_rate)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 290, in heartbeat_check
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return self.transport.heartbeat_check(self.connection, rate=rate)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 149, in heartbeat_check
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return connection.heartbeat_tick(rate=rate)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 718, in heartbeat_tick
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     self.send_heartbeat()
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 669, in send_heartbeat
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     self.frame_writer(8, 0, None, None, None)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 166, in write_frame
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     write(view[:offset])
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 275, in write
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     self._write(s)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 390, in sendall
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     tail = self.send(data, flags)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in send
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return self._send_loop(self.fd.send, data, flags)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in _send_loop
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit     return send_method(data, *args)
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer
2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit 
2020-10-06 15:19:05.696 55 DEBUG oslo.messaging._drivers.impl_rabbit [-] [a2e721c1-7e9d-42d4-af9c-df2f9c2c7322] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:765
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, **kwargs)
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, channel=channels[0], **kwargs), channels[0]
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 816, in execute_method
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit     method()
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1088, in _consume
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit     raise self.connection.recoverable_connection_errors[0]
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit RecoverableConnectionError: <AMQPError: unknown error>
2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit 
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2e721c1-7e9d-42d4-af9c-df2f9c2c7322] AMQP server on overcloud-controller-1:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
2020-10-06 15:19:05.697 55 DEBUG oslo.messaging._drivers.impl_rabbit [-] [80610839-4914-4df7-b1ab-d5d0efcc2fae] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:765
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, **kwargs)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, channel=channels[0], **kwargs), channels[0]
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 816, in execute_method
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     method()
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1103, in _consume
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     self._heartbeat_check()
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 969, in _heartbeat_check
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     self.connection.heartbeat_check(rate=self.heartbeat_rate)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 290, in heartbeat_check
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return self.transport.heartbeat_check(self.connection, rate=rate)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 149, in heartbeat_check
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return connection.heartbeat_tick(rate=rate)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 718, in heartbeat_tick
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     self.send_heartbeat()
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 669, in send_heartbeat
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     self.frame_writer(8, 0, None, None, None)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 166, in write_frame
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     write(view[:offset])
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 275, in write
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     self._write(s)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 390, in sendall
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     tail = self.send(data, flags)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in send
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return self._send_loop(self.fd.send, data, flags)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in _send_loop
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit     return send_method(data, *args)
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer
2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit 
2020-10-06 15:19:05.698 55 ERROR oslo.messaging._drivers.impl_rabbit [-] [80610839-4914-4df7-b1ab-d5d0efcc2fae] AMQP server on overcloud-controller-0:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer



rabbitmq logs:
~~~
These are in the rabbitmq logs

=ERROR REPORT==== 6-Oct-2020::16:55:41 ===
closing AMQP connection <0.20256.0> (10.10.10.10:39952 -> 10.10.10.11:5672 - cinder-volume:55:b8e9ab46-ab18-45e6-835d-78aa1756bd2e):
missed heartbeats from client, timeout: 60s
~~~

Comment 6 Takashi Kajinami 2020-11-03 23:41:45 UTC

*** Bug 1894303 has been marked as a duplicate of this bug. ***

Comment 17 Tzach Shefi 2020-11-18 19:33:09 UTC

Verified on:
openstack-cinder-15.3.1-5.el8ost.noarch


Created a Cinder volume volA, filled it with some data. 
(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --source-volid f135f778-fab3-44d1-901d-22c0e1330919  --name volB+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2020-11-18T19:24:13.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 015b6c15-6134-4e3b-a747-157fc7c85ec4 |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | volB                                 |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 23cfee0f63b44e31a44761d5da636209     |
| replication_status             | None                                 |
| size                           | 1                                    |
| snapshot_id                    | None                                 |
| source_volid                   | f135f778-fab3-44d1-901d-22c0e1330919 |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | cdb4cc954bc14b1db71b5b14fe87fa2c     |
| volume_type                    | tripleo                              |
+--------------------------------+--------------------------------------+

(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --source-volid 015b6c15-6134-4e3b-a747-157fc7c85ec4  --name volC
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2020-11-18T19:25:08.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8 |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | volC                                 |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 23cfee0f63b44e31a44761d5da636209     |
| replication_status             | None                                 |
| size                           | 1                                    |
| snapshot_id                    | None                                 |
| source_volid                   | 015b6c15-6134-4e3b-a747-157fc7c85ec4 |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | cdb4cc954bc14b1db71b5b14fe87fa2c     |
| volume_type                    | tripleo                              |
+--------------------------------+--------------------------------------+


(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --source-volid 2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8  --name volD
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2020-11-18T19:25:34.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | volD                                 |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 23cfee0f63b44e31a44761d5da636209     |
| replication_status             | None                                 |
| size                           | 1                                    |
| snapshot_id                    | None                                 |
| source_volid                   | 2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8 |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | cdb4cc954bc14b1db71b5b14fe87fa2c     |
| volume_type                    | tripleo                              |
+--------------------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --source-volid bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f  --name volE
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2020-11-18T19:25:58.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 73a45018-6664-491a-b754-9c59be267807 |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | volE                                 |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 23cfee0f63b44e31a44761d5da636209     |
| replication_status             | None                                 |
| size                           | 1                                    |
| snapshot_id                    | None                                 |
| source_volid                   | bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | cdb4cc954bc14b1db71b5b14fe87fa2c     |
| volume_type                    | tripleo                              |
+--------------------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --source-volid 73a45018-6664-491a-b754-9c59be267807  --name volF
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2020-11-18T19:26:16.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 3f9356de-e3ab-4d0b-a2c9-7d02bc26a68c |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | volF                                 |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 23cfee0f63b44e31a44761d5da636209     |
| replication_status             | None                                 |
| size                           | 1                                    |
| snapshot_id                    | None                                 |
| source_volid                   | 73a45018-6664-491a-b754-9c59be267807 |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | cdb4cc954bc14b1db71b5b14fe87fa2c     |
| volume_type                    | tripleo                              |
+--------------------------------+--------------------------------------+


Once we passed the depth of 5 clones with volF,we noticed the flatten operation on c-vol.
Also per bz verification no errors or tracebacks show up now, which did happen before the fix.  

C-vol log reports:

2020-11-18 19:22:33.934 7 DEBUG oslo_service.service [req-d64dbea1-34f3-4e6e-a319-ad94e6c21036 - - - - -] backend_defaults.rbd_max_clone_depth = 5 log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2589
[root@controller-2 ~]# grep depth /var/log/containers/cinder/cinder-volume.log
2020-11-18 18:18:56.561 39 INFO cinder.volume.drivers.rbd [req-abc397a1-4c67-4c56-be7f-6456b204876d cdb4cc954bc14b1db71b5b14fe87fa2c 23cfee0f63b44e31a44761d5da636209 - default default] maximum clone depth (1) has been reached - flattening dest volume
2020-11-18 18:20:04.989 39 INFO cinder.volume.drivers.rbd [req-b5fe1c63-69d0-421a-ac6c-068e065bedfb cdb4cc954bc14b1db71b5b14fe87fa2c 23cfee0f63b44e31a44761d5da636209 - default default] maximum clone depth (1) has been reached - flattening dest volume

Flatting works, we have no errors or traceback on c-vol log. 

Ceph backend:

[root@controller-2 /]#  rbd -p volumes ls -l
NAME                                                                                               SIZE  PARENT                                                                                                     FMT PROT LOCK 
volume-015b6c15-6134-4e3b-a747-157fc7c85ec4                                                        1 GiB volumes/volume-f135f778-fab3-44d1-901d-22c0e1330919   2           
volume-015b6c15-6134-4e3b-a747-157fc7c85ec4 1 GiB volumes/volume-f135f778-fab3-44d1-901d-22c0e1330919   2 yes       
volume-0a6b6446-4717-453b-b7f6-c92155665a28.deleted                                                1 GiB                                                                                                              2           
volume-2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8                                                        1 GiB volumes/volume-015b6c15-6134-4e3b-a747-157fc7c85ec4   2           
volume-2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8 1 GiB volumes/volume-015b6c15-6134-4e3b-a747-157fc7c85ec4   2 yes       
volume-3f9356de-e3ab-4d0b-a2c9-7d02bc26a68c                                                        1 GiB volumes/volume-73a45018-6664-491a-b754-9c59be267807   2           
volume-73a45018-6664-491a-b754-9c59be267807                                                        1 GiB volumes/volume-bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f   2           
volume-73a45018-6664-491a-b754-9c59be267807 1 GiB volumes/volume-bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f   2 yes       
volume-bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f                                                        1 GiB volumes/volume-2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8   2           
volume-bc1896a6-6fc0-4f54-a89f-ff0e68ec4b8f 1 GiB volumes/volume-2dbbdd5d-ff22-4fe9-9861-9ca8b39285d8   2 yes       
volume-f135f778-fab3-44d1-901d-22c0e1330919                                                        1 GiB                                                                                                              2           
volume-f135f778-fab3-44d1-901d-22c0e1330919 1 GiB                                                                                                              2 yes

Comment 24 errata-xmlrpc 2020-12-15 18:37:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.3 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:5413

Comment 25 Red Hat Bugzilla 2023-09-15 00:49:37 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days