Bug 1885734
| Summary: | When flattening RBD images, cinder-volume losses connectivity to rabbitmq due to client heartbeat timeout | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | David Hill <dhill> | |
| Component: | openstack-cinder | Assignee: | Eric Harney <eharney> | |
| Status: | CLOSED ERRATA | QA Contact: | Tzach Shefi <tshefi> | |
| Severity: | high | Docs Contact: | Chuck Copello <ccopello> | |
| Priority: | high | |||
| Version: | 13.0 (Queens) | CC: | eharney, fwissing, gfidente, ggrimaux, ltoscano, rsunog, senrique, tkajinam | |
| Target Milestone: | z14 | Keywords: | Triaged, ZStream | |
| Target Release: | 13.0 (Queens) | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-cinder-12.0.10-20.el7ost | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1894303 (view as bug list) | Environment: | ||
| Last Closed: | 2020-12-16 13:57:57 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1888011 | |||
| Bug Blocks: | 1894303 | |||
Red Hat OpenStack Platform release 13.0.12 (Queens) One suspicious thing I found during code investigation is that there are no tpool.Proxy used with the flatten call in create_cloned_volume. Do we need to change the call so that it use tpool.Proxy like the following example ? https://review.opendev.org/#/c/756416/1/cinder/volume/drivers/rbd.py I found that the flatten method called by create_volume_from_snapshot DOES use that tpool.Proxy, so they might be able to workaround the issue by - enable rbd_flatten_volume_from_snapshot in cinder.conf (*) - Create a snapshot from the source volume - Create a new volume from the snapshot (*) I don't know whether this is actually required, but I'll enable this to avoid any dependency problems in the future, and also see whether proxied flatten causes the problem or not. The same issue with volume-from-snapshot has been reported in another bug[1] and fixed by the same strategy (using tpool.Proxy)[2] a while ago, and this shows that we need the same fix for the flatten call in create_cloned_volume... [1] https://bugs.launchpad.net/cinder/+bug/1658037 [2] https://review.opendev.org/#/c/423184/ https://review.opendev.org/#/c/758440/ => ussuri Eric, Per testing, anything I should do other than: rbd_flatten_volume_from_snapshot = true Restart c-vol Create volA from image Create snapshot of volA Create VolB from snapshot If VolB is created we're good. Don't see anything per vol/snap sizes do these have any impact on the timeouts? I plan on using Cirros or a rhel 7 image, would this be enough? Should I try "rbd_max_clone_depth to 1" as well? Or is that relevant only bz1889894, in which case for this verification I can ignore this. Verified on:
openstack-cinder-12.0.10-20.el7ost.noarch
On a deployment backed by ceph, I've set this on cinder.conf:
rbd_flatten_volume_from_snapshot = true
Uploaded raw image, created vol A from it:
(overcloud) [stack@undercloud-0 ~]$ cinder create 5 --name volA --image cirrosraw
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-11-11T12:44:46.000000 |
| description | None |
| encrypted | False |
| id | 2ba22f49-f2d2-4645-871c-91564e8bf1d8 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | volA |
| os-vol-host-attr:host | hostgroup@tripleo_ceph#tripleo_ceph |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | a5bff9312f4c48f4a7e68ecf25895715 |
| replication_status | None |
| size | 5 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | 2020-11-11T12:44:46.000000 |
| user_id | 5142c97480044e3ab57fefc864b0b403 |
| volume_type | tripleo |
+--------------------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| 2ba22f49-f2d2-4645-871c-91564e8bf1d8 | available | volA | 5 | tripleo | true | |
Create snapshot of volA:
(overcloud) [stack@undercloud-0 ~]$ cinder snapshot-create 2ba22f49-f2d2-4645-871c-91564e8bf1d8 --name snapA
+-------------+--------------------------------------+
| Property | Value |
+-------------+--------------------------------------+
| created_at | 2020-11-11T12:46:02.745924 |
| description | None |
| id | 86572e77-ddd3-4114-891c-649e008fcab7 |
| metadata | {} |
| name | snapA |
| size | 5 |
| status | creating |
| updated_at | None |
| volume_id | 2ba22f49-f2d2-4645-871c-91564e8bf1d8 |
+-------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ cinder snapshot-list
+--------------------------------------+--------------------------------------+-----------+-------+------+
| ID | Volume ID | Status | Name | Size |
+--------------------------------------+--------------------------------------+-----------+-------+------+
| 86572e77-ddd3-4114-891c-649e008fcab7 | 2ba22f49-f2d2-4645-871c-91564e8bf1d8 | available | snapA | 5 |
+--------------------------------------+--------------------------------------+-----------+-------+------+
Use said snapshot to create volB:
(overcloud) [stack@undercloud-0 ~]$ cinder create 5 --snapshot-id 86572e77-ddd3-4114-891c-649e008fcab7 --name volB
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-11-11T12:48:54.000000 |
| description | None |
| encrypted | False |
| id | ac0ab474-ede8-482d-bf4f-374cb9420786 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | volB |
| os-vol-host-attr:host | hostgroup@tripleo_ceph#tripleo_ceph |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | a5bff9312f4c48f4a7e68ecf25895715 |
| replication_status | None |
| size | 5 |
| snapshot_id | 86572e77-ddd3-4114-891c-649e008fcab7 |
| source_volid | None |
| status | creating |
| updated_at | 2020-11-11T12:48:54.000000 |
| user_id | 5142c97480044e3ab57fefc864b0b403 |
| volume_type | tripleo |
+--------------------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| 2ba22f49-f2d2-4645-871c-91564e8bf1d8 | available | volA | 5 | tripleo | true | |
| ac0ab474-ede8-482d-bf4f-374cb9420786 | available | volB | 5 | tripleo | true | |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ cinder show volB
+--------------------------------+-------------------------------------------------+
| Property | Value |
+--------------------------------+-------------------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2020-11-11T12:48:54.000000 |
| description | None |
| encrypted | False |
| id | ac0ab474-ede8-482d-bf4f-374cb9420786 |
| metadata | |
| migration_status | None |
| multiattach | False |
| name | volB |
| os-vol-host-attr:host | hostgroup@tripleo_ceph#tripleo_ceph |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | a5bff9312f4c48f4a7e68ecf25895715 |
| replication_status | None |
| size | 5 |
| snapshot_id | 86572e77-ddd3-4114-891c-649e008fcab7 |
| source_volid | None |
| status | available |
| updated_at | 2020-11-11T12:48:56.000000 |
| user_id | 5142c97480044e3ab57fefc864b0b403 |
| volume_image_metadata | checksum : ba3cd24377dde5dfdd58728894004abb |
| | container_format : bare |
| | disk_format : raw |
| | image_id : 90be0bc4-c760-4c60-851c-2ac34765293c |
| | image_name : cirrosraw |
| | min_disk : 0 |
| | min_ram : 0 |
| | size : 46137344 |
| volume_type | tripleo |
+--------------------------------+-------------------------------------------------+
A cloned volume was successfully created from a snapshot, with a none default setting of: rbd_flatten_volume_from_snapshot = true
I may read this incorrectly, but isn't this the verification of the workaround? I'm afraid that Luigi is correct. The issue was observed when creating a volume "from a volume", and was not reproduced when creating a volume "from a snapshot". # That is why we suggested creating a volume from a snapshot as workaround. Also, please make sure that you create multiple cloned volumes, since cinder doesn't try flatten until the source volume reaches rbd_max_clone_depth . (You can decrease the parameter from its default 5, if needed) Thank you for correcting me Luigi and Takashi,
Lets revisit this again, re-tested on:
openstack-cinder-12.0.10-21.el7ost.noarch
Created a Cinder volume, mounted and filled with random data.
(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 12f63e6b-fd35-4679-af2c-7cd3c7fb7dee | in-use | volA | 2 | tripleo | false | abc1a703-8d3d-47a8-ad81-7ddddac6fb12 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
Now lets clone this volA to a new volume
(overcloud) [stack@undercloud-0 ~]$ cinder create 2 --name volB_clone1 --source-volid 12f63e6b-fd35-4679-af2c-7cd3c7fb7dee
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-11-18T14:50:47.000000 |
| description | None |
| encrypted | False |
| id | 34bff18c-ed4d-41b2-8f63-162c5486e6ef |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | volB_clone1 |
| os-vol-host-attr:host | hostgroup@tripleo_ceph#tripleo_ceph |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | fe7c9e7e23104b839fef51ea7e00d1c8 |
| replication_status | None |
| size | 2 |
| snapshot_id | None |
| source_volid | 12f63e6b-fd35-4679-af2c-7cd3c7fb7dee |
| status | creating |
| updated_at | 2020-11-18T14:50:48.000000 |
| user_id | 3170f966a536448abc8b20d69fa3c98d |
| volume_type | tripleo |
+--------------------------------+--------------------------------------+
[root@controller-0 ~]# rbd -p volumes ls -l
NAME SIZE PARENT FMT PROT LOCK
volume-12f63e6b-fd35-4679-af2c-7cd3c7fb7dee 2GiB 2
volume-12f63e6b-fd35-4679-af2c-7cd3c7fb7dee 2GiB 2 yes
volume-34bff18c-ed4d-41b2-8f63-162c5486e6ef 2GiB volumes/volume-12f63e6b-fd35-4679-af2c-7cd3c7fb7dee 2
We see original volA plus it's .clone.snap
And the volB_clone1
Re-clone again a second time.
(overcloud) [stack@undercloud-0 ~]$ cinder create 2 --name volC_clone2 --source-volid 34bff18c-ed4d-41b2-8f63-162c5486e6ef
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-11-18T14:54:08.000000 |
| description | None |
| encrypted | False |
| id | d304aa90-0409-4508-bf81-306a9f686f90 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | volC_clone2 |
| os-vol-host-attr:host | hostgroup@tripleo_ceph#tripleo_ceph |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | fe7c9e7e23104b839fef51ea7e00d1c8 |
| replication_status | None |
| size | 2 |
| snapshot_id | None |
| source_volid | 34bff18c-ed4d-41b2-8f63-162c5486e6ef |
| status | creating |
| updated_at | 2020-11-18T14:54:09.000000 |
| user_id | 3170f966a536448abc8b20d69fa3c98d |
| volume_type | tripleo |
+--------------------------------+--------------------------------------+
Re-clone 3rd time
(overcloud) [stack@undercloud-0 ~]$ cinder create 2 --name volD_clone3 --source-volid d304aa90-0409-4508-bf81-306a9f686f90
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-11-18T14:57:16.000000 |
| description | None |
| encrypted | False |
| id | ca84ca86-bbcd-4f1d-935d-b44f364769b8 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | volD_clone3 |
| os-vol-host-attr:host | hostgroup@tripleo_ceph#tripleo_ceph |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | fe7c9e7e23104b839fef51ea7e00d1c8 |
| replication_status | None |
| size | 2 |
| snapshot_id | None |
| source_volid | d304aa90-0409-4508-bf81-306a9f686f90 |
| status | creating |
| updated_at | 2020-11-18T14:57:17.000000 |
| user_id | 3170f966a536448abc8b20d69fa3c98d |
| volume_type | tripleo |
+--------------------------------+--------------------------------------+
Re-clone 4th time
(overcloud) [stack@undercloud-0 ~]$ cinder create 2 --name volE_clone4 --source-volid ca84ca86-bbcd-4f1d-935d-b44f364769b8
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-11-18T14:58:06.000000 |
| description | None |
| encrypted | False |
| id | 839194d8-5d50-415d-8466-cf85cf189d22 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | volE_clone4 |
| os-vol-host-attr:host | hostgroup@tripleo_ceph#tripleo_ceph |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | fe7c9e7e23104b839fef51ea7e00d1c8 |
| replication_status | None |
| size | 2 |
| snapshot_id | None |
| source_volid | ca84ca86-bbcd-4f1d-935d-b44f364769b8 |
| status | creating |
| updated_at | 2020-11-18T14:58:06.000000 |
| user_id | 3170f966a536448abc8b20d69fa3c98d |
| volume_type | tripleo |
+--------------------------------+--------------------------------------+
Re-clone 5th time
(overcloud) [stack@undercloud-0 ~]$ cinder create 2 --name volF_clone5 --source-volid 839194d8-5d50-415d-8466-cf85cf189d22
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-11-18T14:58:37.000000 |
| description | None |
| encrypted | False |
| id | fbdd6c64-c121-45ab-b63e-00eed3996061 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | volF_clone5 |
| os-vol-host-attr:host | hostgroup@tripleo_ceph#tripleo_ceph |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | fe7c9e7e23104b839fef51ea7e00d1c8 |
| replication_status | None |
| size | 2 |
| snapshot_id | None |
| source_volid | 839194d8-5d50-415d-8466-cf85cf189d22 |
| status | creating |
| updated_at | 2020-11-18T14:58:37.000000 |
| user_id | 3170f966a536448abc8b20d69fa3c98d |
| volume_type | tripleo |
+--------------------------------+--------------------------------------+
We have a depth of 5 clones, lets check Ceph side:
[root@controller-0 ~]# rbd -p volumes ls -l
NAME SIZE PARENT FMT PROT LOCK
volume-12f63e6b-fd35-4679-af2c-7cd3c7fb7dee (volA) 2GiB 2
volume-12f63e6b-fd35-4679-af2c-7cd3c7fb7dee 2GiB 2 yes
volume-34bff18c-ed4d-41b2-8f63-162c5486e6ef (volB_clone1) 2GiB volumes/volume-12f63e6b-fd35-4679-af2c-7cd3c7fb7dee 2
volume-34bff18c-ed4d-41b2-8f63-162c5486e6ef 2GiB volumes/volume-12f63e6b-fd35-4679-af2c-7cd3c7fb7dee 2 yes
volume-d304aa90-0409-4508-bf81-306a9f686f90 (volC_clone2) 2GiB volumes/volume-34bff18c-ed4d-41b2-8f63-162c5486e6ef 2
volume-d304aa90-0409-4508-bf81-306a9f686f90 2GiB volumes/volume-34bff18c-ed4d-41b2-8f63-162c5486e6ef 2 yes
volume-ca84ca86-bbcd-4f1d-935d-b44f364769b8 (volD_clone3) 2GiB volumes/volume-d304aa90-0409-4508-bf81-306a9f686f90 2
volume-ca84ca86-bbcd-4f1d-935d-b44f364769b8 2GiB volumes/volume-d304aa90-0409-4508-bf81-306a9f686f90 2 yes
volume-839194d8-5d50-415d-8466-cf85cf189d22 (volE_clone4) 2GiB volumes/volume-ca84ca86-bbcd-4f1d-935d-b44f364769b8 2
volume-839194d8-5d50-415d-8466-cf85cf189d22 2GiB volumes/volume-ca84ca86-bbcd-4f1d-935d-b44f364769b8 2 yes
volume-fbdd6c64-c121-45ab-b63e-00eed3996061 (volF_clone5) 2GiB volumes/volume-839194d8-5d50-415d-8466-cf85cf189d22 2
Still I didn't notice any flatting notices on logs.
Which is legit as I didn't pass the max depth of 5 (yet)
Thus I re-cloned a 6th time.
(overcloud) [stack@undercloud-0 ~]$ cinder create 2 --name volG_clone6 --source-volid fbdd6c64-c121-45ab-b63e-00eed3996061
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-11-18T15:42:15.000000 |
| description | None |
| encrypted | False |
| id | 01f71541-797e-48bf-8121-739c63e15cbd |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | volG_clone6 |
| os-vol-host-attr:host | hostgroup@tripleo_ceph#tripleo_ceph |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | fe7c9e7e23104b839fef51ea7e00d1c8 |
| replication_status | None |
| size | 2 |
| snapshot_id | None |
| source_volid | fbdd6c64-c121-45ab-b63e-00eed3996061 |
| status | creating |
| updated_at | 2020-11-18T15:42:16.000000 |
| user_id | 3170f966a536448abc8b20d69fa3c98d |
| volume_type | tripleo |
+--------------------------------+--------------------------------------+
Which triggered the expected flatting:
/var/log/containers/cinder/cinder-volume.log:4406:2020-11-18 15:42:17.516 61 INFO cinder.volume.drivers.rbd [req-37251d32-00c0-42fb-ad07-62bdbe168b40 3170f966a536448abc8b20d69fa3c98d fe7c9e7e23104b839fef51ea7e00d1c8 - default default] maximum clone depth (5) has been reached - flattening dest volume
4405 2020-11-18 15:42:17.513 61 DEBUG cinder.volume.drivers.rbd [req-37251d32-00c0-42fb-ad07-62bdbe168b40 3170f966a536448abc8b20d69fa3c98d fe7c9e7e23104b839fef51ea7e00d1c8 - default default] Volume volume-12f63e6b-fd35-4679-af2c-7cd3c 7fb7dee is not a clone. _get_clone_info /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:968
4406 2020-11-18 15:42:17.516 61 INFO cinder.volume.drivers.rbd [req-37251d32-00c0-42fb-ad07-62bdbe168b40 3170f966a536448abc8b20d69fa3c98d fe7c9e7e23104b839fef51ea7e00d1c8 - default default] maximum clone depth (5) has been reached - f lattening dest volume
4407 2020-11-18 15:42:17.670 61 DEBUG cinder.volume.drivers.rbd [req-37251d32-00c0-42fb-ad07-62bdbe168b40 3170f966a536448abc8b20d69fa3c98d fe7c9e7e23104b839fef51ea7e00d1c8 - default default] flattening dest volume volume-01f71541-797e -48bf-8121-739c63e15cbd create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:623
4408 2020-11-18 15:42:22.300 61 DEBUG cinder.volume.drivers.rbd [req-37251d32-00c0-42fb-ad07-62bdbe168b40 3170f966a536448abc8b20d69fa3c98d fe7c9e7e23104b839fef51ea7e00d1c8 - default default] remove temporary snap volume-01f71541-797e- 48bf-8121-739c63e15cbd.clone_snap create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:636
4409 2020-11-18 15:42:22.426 61 DEBUG cinder.volume.drivers.rbd [req-37251d32-00c0-42fb-ad07-62bdbe168b40 3170f966a536448abc8b20d69fa3c98d fe7c9e7e23104b839fef51ea7e00d1c8 - default default] clone created successfully create_cloned_vo lume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:662
And we also notice the last volume volG_clone6 who's ID ends with cbd isn't x.clone_snap from it's source.
[root@controller-0 ~]# rbd -p volumes ls -l
NAME SIZE PARENT FMT PROT LOCK
volume-01f71541-797e-48bf-8121-739c63e15cbd 2GiB 2
volume-12f63e6b-fd35-4679-af2c-7cd3c7fb7dee 2GiB 2
volume-12f63e6b-fd35-4679-af2c-7cd3c7fb7dee 2GiB 2 yes
volume-34bff18c-ed4d-41b2-8f63-162c5486e6ef 2GiB volumes/volume-12f63e6b-fd35-4679-af2c-7cd3c7fb7dee 2
volume-34bff18c-ed4d-41b2-8f63-162c5486e6ef 2GiB volumes/volume-12f63e6b-fd35-4679-af2c-7cd3c7fb7dee 2 yes
volume-839194d8-5d50-415d-8466-cf85cf189d22 2GiB volumes/volume-ca84ca86-bbcd-4f1d-935d-b44f364769b8 2
volume-839194d8-5d50-415d-8466-cf85cf189d22 2GiB volumes/volume-ca84ca86-bbcd-4f1d-935d-b44f364769b8 2 yes
volume-ca84ca86-bbcd-4f1d-935d-b44f364769b8 2GiB volumes/volume-d304aa90-0409-4508-bf81-306a9f686f90 2
volume-ca84ca86-bbcd-4f1d-935d-b44f364769b8 2GiB volumes/volume-d304aa90-0409-4508-bf81-306a9f686f90 2 yes
volume-d304aa90-0409-4508-bf81-306a9f686f90 2GiB volumes/volume-34bff18c-ed4d-41b2-8f63-162c5486e6ef 2
volume-d304aa90-0409-4508-bf81-306a9f686f90 2GiB volumes/volume-34bff18c-ed4d-41b2-8f63-162c5486e6ef 2 yes
volume-fbdd6c64-c121-45ab-b63e-00eed3996061
Didn't hit any errors or traceback during last operation or it's flatting.
Good to verify, this time I hope done correctly.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (openstack-cinder bug fix advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:5579 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |
Description of problem: When flattening RBD images, cinder-volume losses connectivity to rabbitmq due to client heartbeat timeout and subsequent volume creation appears to be failing: 2020-10-06 15:15:50.746 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:15:50.846 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] creating snapshot='volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72.clone_snap' create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:588 2020-10-06 15:15:51.585 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] cloning 'volume-d5b6f370-11b9-4561-b8e6-ec40bf049cca' to 'volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72' create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:597 2020-10-06 15:15:52.003 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] Volume volume-eb0c640e-f22e-4b49-8675-cff8bc39ea26.deleted is not a clone. _get_clone_info /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:972 2020-10-06 15:15:52.005 55 INFO cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] maximum clone depth (5) has been reached - flattening dest volume 2020-10-06 15:15:52.091 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] flattening dest volume volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72 create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:625 2020-10-06 15:15:55.586 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:15:55.619 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:15:55.656 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194 2020-10-06 15:16:41.979 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:16:42.017 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:16:42.052 110 DEBUG cinder.manager [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194 2020-10-06 15:16:55.591 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:16:55.623 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:16:55.661 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194 2020-10-06 15:17:41.976 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:17:42.012 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:17:42.047 110 DEBUG cinder.manager [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194 2020-10-06 15:17:55.586 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:17:55.623 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:17:55.666 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194 2020-10-06 15:18:41.981 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:18:42.016 110 DEBUG cinder.volume.drivers.rbd [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:18:42.049 110 DEBUG cinder.manager [req-2e9c0e29-42cd-4d18-b0a3-f8e0517fa367 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194 2020-10-06 15:18:55.591 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:18:55.624 70 DEBUG cinder.volume.drivers.rbd [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-10-06 15:18:55.657 70 DEBUG cinder.manager [req-06296211-f754-401c-8cdf-c6b65fe61d25 - - - - -] Notifying Schedulers of capabilities ... _publish_service_capabilities /usr/lib/python2.7/site-packages/cinder/manager.py:194 2020-10-06 15:19:05.575 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] remove temporary snap volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72.clone_snap create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:640 2020-10-06 15:19:05.688 55 DEBUG cinder.volume.drivers.rbd [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] clone created successfully create_cloned_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:666 2020-10-06 15:19:05.690 55 DEBUG cinder.volume.manager [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (5b07b48e-e622-4603-b45b-81ac5099ea67) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'status': u'creating', 'volume_size': 200, 'volume_name': u'volume-de8d188a-ebcb-422d-b76d-3fad2e0d3b72', 'source_volid': u'd5b6f370-11b9-4561-b8e6-ec40bf049cca', 'source_volstatus': u'in-use'}' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183 2020-10-06 15:19:05.691 55 DEBUG cinder.volume.manager [req-2bb6029d-be03-4aab-9a0e-95cec31c4dd8 e54fa6be51954cf9b0b2863023d69b27 8ecf6e04b50e4954832784c3c2794d2a - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeOnFinishTask;volume:create, create.end' (3addaf14-ccd7-4889-8dc0-1b2b4d880911) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 2020-10-06 15:19:05.694 55 DEBUG oslo.messaging._drivers.impl_rabbit [-] [a2e721c1-7e9d-42d4-af9c-df2f9c2c7322] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:765 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__ 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0] 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 816, in execute_method 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit method() 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1103, in _consume 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit self._heartbeat_check() 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 969, in _heartbeat_check 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit self.connection.heartbeat_check(rate=self.heartbeat_rate) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 290, in heartbeat_check 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit return self.transport.heartbeat_check(self.connection, rate=rate) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 149, in heartbeat_check 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit return connection.heartbeat_tick(rate=rate) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 718, in heartbeat_tick 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit self.send_heartbeat() 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 669, in send_heartbeat 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit self.frame_writer(8, 0, None, None, None) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 166, in write_frame 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit write(view[:offset]) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 275, in write 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit self._write(s) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 390, in sendall 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in send 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit return self._send_loop(self.fd.send, data, flags) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in _send_loop 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit return send_method(data, *args) 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer 2020-10-06 15:19:05.694 55 ERROR oslo.messaging._drivers.impl_rabbit 2020-10-06 15:19:05.696 55 DEBUG oslo.messaging._drivers.impl_rabbit [-] [a2e721c1-7e9d-42d4-af9c-df2f9c2c7322] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:765 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs) 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__ 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0] 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 816, in execute_method 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit method() 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1088, in _consume 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit raise self.connection.recoverable_connection_errors[0] 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit RecoverableConnectionError: <AMQPError: unknown error> 2020-10-06 15:19:05.696 55 ERROR oslo.messaging._drivers.impl_rabbit 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit [-] [a2e721c1-7e9d-42d4-af9c-df2f9c2c7322] AMQP server on overcloud-controller-1:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error> 2020-10-06 15:19:05.697 55 DEBUG oslo.messaging._drivers.impl_rabbit [-] [80610839-4914-4df7-b1ab-d5d0efcc2fae] Received recoverable error from kombu: on_error /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:765 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__ 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0] 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 816, in execute_method 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit method() 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1103, in _consume 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit self._heartbeat_check() 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 969, in _heartbeat_check 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit self.connection.heartbeat_check(rate=self.heartbeat_rate) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 290, in heartbeat_check 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit return self.transport.heartbeat_check(self.connection, rate=rate) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 149, in heartbeat_check 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit return connection.heartbeat_tick(rate=rate) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 718, in heartbeat_tick 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit self.send_heartbeat() 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 669, in send_heartbeat 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit self.frame_writer(8, 0, None, None, None) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 166, in write_frame 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit write(view[:offset]) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 275, in write 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit self._write(s) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 390, in sendall 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 384, in send 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit return self._send_loop(self.fd.send, data, flags) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 371, in _send_loop 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit return send_method(data, *args) 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer 2020-10-06 15:19:05.697 55 ERROR oslo.messaging._drivers.impl_rabbit 2020-10-06 15:19:05.698 55 ERROR oslo.messaging._drivers.impl_rabbit [-] [80610839-4914-4df7-b1ab-d5d0efcc2fae] AMQP server on overcloud-controller-0:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer rabbitmq logs: ~~~ These are in the rabbitmq logs =ERROR REPORT==== 6-Oct-2020::16:55:41 === closing AMQP connection <0.20256.0> (10.10.10.10:39952 -> 10.10.10.11:5672 - cinder-volume:55:b8e9ab46-ab18-45e6-835d-78aa1756bd2e): missed heartbeats from client, timeout: 60s ~~~ Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: