Bug 1892995
| Summary: | cinder volume failing when setting rbd_max_clone_depth=1 | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Eric Harney <eharney> |
| Component: | openstack-cinder | Assignee: | Eric Harney <eharney> |
| Status: | CLOSED ERRATA | QA Contact: | Tzach Shefi <tshefi> |
| Severity: | medium | Docs Contact: | RHOS Documentation Team <rhos-docs> |
| Priority: | medium | ||
| Version: | 16.2 (Train) | CC: | jmelvin |
| Target Milestone: | beta | Keywords: | Triaged |
| Target Release: | 16.2 (Train on RHEL 8.4) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-cinder-15.4.2-2.20210209134940.abcefa3.el8ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-09-15 07:09:53 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1889894 | ||
|
Description
Eric Harney
2020-10-29 19:35:00 UTC
Verified on:
openstack-cinder-15.4.2-2.20210225045015
Created an empty volume:
(overcloud) [stack@undercloud-0 ~]$ cinder create 2 --name volA
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-03-31T10:27:07.000000 |
| description | None |
| encrypted | False |
| id | 67c271f4-a180-42f5-8c2f-8a06781668e0 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | volA |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 7e21ac2b38824e6a9b4c686baed65772 |
| replication_status | None |
| size | 2 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | b000f0a966e84c14affe64e43b37a2d8 |
| volume_type | tripleo |
+--------------------------------+--------------------------------------+
Attached to instance filled with data and then detach.
Cloned over again and again
(overcloud) [stack@undercloud-0 ~]$ cinder create 2 --source-volid 67c271f4-a180-42f5-8c2f-8a06781668e0 --name volB
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-03-31T10:37:33.000000 |
| description | None |
| encrypted | False |
| id | f1d503f8-9398-4f4a-9a3e-cee4e7c431f9 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | volB |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 7e21ac2b38824e6a9b4c686baed65772 |
| replication_status | None |
| size | 2 |
| snapshot_id | None |
| source_volid | 67c271f4-a180-42f5-8c2f-8a06781668e0 |
| status | creating |
| updated_at | None |
| user_id | b000f0a966e84c14affe64e43b37a2d8 |
| volume_type | tripleo |
+--------------------------------+--------------------------------------+
Till we have a list of 7 volumes, each one a clone of the volume before it.
(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 07aa9608-64d0-43f6-a232-da6524de6309 | available | volG | 2 | tripleo | false | |
| 1cfa6578-ba38-48d1-bbf0-149b07bd6c9c | available | volF | 2 | tripleo | false | |
| 1f2b5ab1-0ddd-474c-95ea-a511562f4f74 | available | volC | 2 | tripleo | false | |
| 67c271f4-a180-42f5-8c2f-8a06781668e0 | available | volA | 2 | tripleo | false | |
| 8355b998-714c-474d-bd5a-2581eaf21ef8 | available | volE | 2 | tripleo | false | |
| be05e68c-a2b9-4f5f-a835-f563e1a89138 | available | volD | 2 | tripleo | false | |
| f1d503f8-9398-4f4a-9a3e-cee4e7c431f9 | available | volB | 2 | tripleo | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
All this was done with the default of
[root@controller-0 ~]# grep -irn rbd_max /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf
2990:#rbd_max_clone_depth = 5
Now lets kick it down to 1 and restart c-vol
rbd_max_clone_depth = 1
And check c-vol log to see if we hit any error, nothing yet.
Lets clone another volume again after setting max depth to 1
(overcloud) [stack@undercloud-0 ~]$ cinder create 2 --source-volid 07aa9608-64d0-43f6-a232-da6524de6309 --name volH
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-03-31T10:46:09.000000 |
| description | None |
| encrypted | False |
| id | 7e5d9fda-f406-42d0-8f3d-b5c506524e16 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | volH |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 7e21ac2b38824e6a9b4c686baed65772 |
| replication_status | None |
| size | 2 |
| snapshot_id | None |
| source_volid | 07aa9608-64d0-43f6-a232-da6524de6309 |
| status | creating |
| updated_at | None |
| user_id | b000f0a966e84c14affe64e43b37a2d8 |
| volume_type | tripleo |
+--------------------------------+--------------------------------------+
All volumes OK:
(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 07aa9608-64d0-43f6-a232-da6524de6309 | available | volG | 2 | tripleo | false | |
| 1cfa6578-ba38-48d1-bbf0-149b07bd6c9c | available | volF | 2 | tripleo | false | |
| 1f2b5ab1-0ddd-474c-95ea-a511562f4f74 | available | volC | 2 | tripleo | false | |
| 67c271f4-a180-42f5-8c2f-8a06781668e0 | available | volA | 2 | tripleo | false | |
| 7e5d9fda-f406-42d0-8f3d-b5c506524e16 | available | volH | 2 | tripleo | false | |
| 8355b998-714c-474d-bd5a-2581eaf21ef8 | available | volE | 2 | tripleo | false | |
| be05e68c-a2b9-4f5f-a835-f563e1a89138 | available | volD | 2 | tripleo | false | |
| f1d503f8-9398-4f4a-9a3e-cee4e7c431f9 | available | volB | 2 | tripleo | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
On c-vol log we see the last two cases where max depth was till 5 and after it was changed to 1:
_defaults.rbd_max_clone_depth = 5 log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2589
1101:2021-03-31 10:39:32.046 46 INFO cinder.volume.drivers.rbd [req-4a06ebcc-5a34-4e74-bc47-02d5c96414eb b000f0a966e84c14affe64e43b37a2d8 7e21ac2b38824e6a9b4c686baed65772 - default default] maximum clone depth (5) has been reached - flattening dest volume
1519:2021-03-31 10:45:29.102 8 DEBUG oslo_service.service [req-e548242d-e416-4730-a67a-fefc36d0aed0 - - - - -] tripleo_ceph.rbd_max_clone_depth = None log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2589
1662:2021-03-31 10:45:29.133 8 DEBUG oslo_service.service [req-e548242d-e416-4730-a67a-fefc36d0aed0 - - - - -] backend_defaults.rbd_max_clone_depth = 1 log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2589
Another test lets create a new volume from an image:
(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --name Newvol --image 680ef9d5-e7ec-4f6d-9927-e0ba636fe4eb
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-03-31T10:49:00.000000 |
| description | None |
| encrypted | False |
| id | b6943cb3-2873-4af3-b762-7db4c791a85f |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | Newvol |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 7e21ac2b38824e6a9b4c686baed65772 |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | b000f0a966e84c14affe64e43b37a2d8 |
| volume_type | tripleo |
+--------------------------------+--------------------------------------+
And now lets clone this new volume:
(overcloud) [stack@undercloud-0 ~]$ cinder create 2 --source-volid b6943cb3-2873-4af3-b762-7db4c791a85f --name NewCloned
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2021-03-31T10:49:55.000000 |
| description | None |
| encrypted | False |
| id | dea3fae7-27a1-4f52-ad7e-a632f8332efa |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | NewCloned |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 7e21ac2b38824e6a9b4c686baed65772 |
| replication_status | None |
| size | 2 |
| snapshot_id | None |
| source_volid | b6943cb3-2873-4af3-b762-7db4c791a85f |
| status | creating |
| updated_at | None |
| user_id | b000f0a966e84c14affe64e43b37a2d8 |
| volume_type | tripleo |
+--------------------------------+--------------------------------------+
Again now errors on c-vol log all volumes are avaliable and we notice the depth reached for 1 too:
_ceph.rbd_max_clone_depth = None log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2589
516:2021-03-31 09:35:39.681 7 DEBUG oslo_service.service [req-4e959c96-362b-413b-9dfa-f08104d3726f - - - - -] backend_defaults.rbd_max_clone_depth = 5 log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2589
1101:2021-03-31 10:39:32.046 46 INFO cinder.volume.drivers.rbd [req-4a06ebcc-5a34-4e74-bc47-02d5c96414eb b000f0a966e84c14affe64e43b37a2d8 7e21ac2b38824e6a9b4c686baed65772 - default default] maximum clone depth (5) has been reached - flattening dest volume
1519:2021-03-31 10:45:29.102 8 DEBUG oslo_service.service [req-e548242d-e416-4730-a67a-fefc36d0aed0 - - - - -] tripleo_ceph.rbd_max_clone_depth = None log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2589
1662:2021-03-31 10:45:29.133 8 DEBUG oslo_service.service [req-e548242d-e416-4730-a67a-fefc36d0aed0 - - - - -] backend_defaults.rbd_max_clone_depth = 1 log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2589
1905:2021-03-31 10:53:42.099 26 INFO cinder.volume.drivers.rbd [req-5c965ea0-1875-4eb6-a8c5-0351c6e7ba82 b000f0a966e84c14affe64e43b37a2d8 7e21ac2b38824e6a9b4c686baed65772 - default default] maximum clone depth (1) has been reached - flattening dest volume
Good to verify.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:3483 |