This bug was initially created as a copy of Bug #1889894 Description of problem: When cinder.conf:rbd_max_clone_depth=1 cinder volume fails cinder.volume.manager does not appear to have the capability to use an updated 'smaller' value of "rbd_max_clone_depth". I'm not seeing a condition calling flatten(for existing RBDs exceeding new clone_depth). In this instance, customer updated rbd_max_clone_depth = 1 # default: 5 REF: https://docs.openstack.org/cinder/queens/sample_config.html # Maximum number of nested volume clones that are taken before a flatten # occurs. Set to 0 to disable cloning. (integer value) #rbd_max_clone_depth = 5 Version-Release number of selected component (if applicable): Distro: [redhat-release] Red Hat Enterprise Linux Server release 7.8 (Maipo) [rhosp-release] Red Hat OpenStack Platform release 13.0.12 (Queens) [os-release] Red Hat Cloud Infrastructure 7.8 (Maipo) openstack-cinder-12.0.10-11.el7ost.noarch Wed Jul 22 17:30:43 2020 puppet-cinder-12.4.1-7.el7ost.noarch Wed Jul 22 17:29:05 2020 python2-cinderclient-3.5.0-2.el7ost.noarch Wed Jul 22 17:29:19 2020 python-cinder-12.0.10-11.el7ost.noarch Wed Jul 22 17:30:26 2020 How reproducible: Change #rbd_max_clone_depth = 5 to rbd_max_clone_depth = 1 Actual results: ~~~ 2020-10-20 11:22:33.012 55 ERROR cinder.volume.manager Traceback (most recent call last): 2020-10-20 11:22:33.012 55 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2020-10-20 11:22:33.012 55 ERROR cinder.volume.manager result = task.execute(**arguments) 2020-10-20 11:22:33.012 55 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 1034, in execute 2020-10-20 11:22:33.012 55 ERROR cinder.volume.manager context, volume, **volume_spec) 2020-10-20 11:22:33.012 55 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 492, in _create_from_source_volume 2020-10-20 11:22:33.012 55 ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref) 2020-10-20 11:22:33.012 55 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 614, in create_cloned_volume 2020-10-20 11:22:33.012 55 ERROR cinder.volume.manager depth = self._get_clone_depth(client, src_name) 2020-10-20 11:22:33.012 55 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 548, in _get_clone_depth 2020-10-20 11:22:33.012 55 ERROR cinder.volume.manager return self._get_clone_depth(client, parent, depth + 1) 2020-10-20 11:22:33.012 55 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 548, in _get_clone_depth 2020-10-20 11:22:33.012 55 ERROR cinder.volume.manager return self._get_clone_depth(client, parent, depth + 1) 2020-10-20 11:22:33.012 55 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 546, in _get_clone_depth 2020-10-20 11:22:33.012 55 ERROR cinder.volume.manager (self.configuration.rbd_max_clone_depth)) 2020-10-20 11:22:33.012 55 ERROR cinder.volume.manager Exception: clone depth exceeds limit of 1 Expected results: operation will occur w/o creating an exception in cinder volume manager.
Verified on: openstack-cinder-15.4.2-2.20210225045015 Created an empty volume: (overcloud) [stack@undercloud-0 ~]$ cinder create 2 --name volA +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2021-03-31T10:27:07.000000 | | description | None | | encrypted | False | | id | 67c271f4-a180-42f5-8c2f-8a06781668e0 | | metadata | {} | | migration_status | None | | multiattach | False | | name | volA | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 7e21ac2b38824e6a9b4c686baed65772 | | replication_status | None | | size | 2 | | snapshot_id | None | | source_volid | None | | status | creating | | updated_at | None | | user_id | b000f0a966e84c14affe64e43b37a2d8 | | volume_type | tripleo | +--------------------------------+--------------------------------------+ Attached to instance filled with data and then detach. Cloned over again and again (overcloud) [stack@undercloud-0 ~]$ cinder create 2 --source-volid 67c271f4-a180-42f5-8c2f-8a06781668e0 --name volB +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2021-03-31T10:37:33.000000 | | description | None | | encrypted | False | | id | f1d503f8-9398-4f4a-9a3e-cee4e7c431f9 | | metadata | {} | | migration_status | None | | multiattach | False | | name | volB | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 7e21ac2b38824e6a9b4c686baed65772 | | replication_status | None | | size | 2 | | snapshot_id | None | | source_volid | 67c271f4-a180-42f5-8c2f-8a06781668e0 | | status | creating | | updated_at | None | | user_id | b000f0a966e84c14affe64e43b37a2d8 | | volume_type | tripleo | +--------------------------------+--------------------------------------+ Till we have a list of 7 volumes, each one a clone of the volume before it. (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------+------+-------------+----------+-------------+ | 07aa9608-64d0-43f6-a232-da6524de6309 | available | volG | 2 | tripleo | false | | | 1cfa6578-ba38-48d1-bbf0-149b07bd6c9c | available | volF | 2 | tripleo | false | | | 1f2b5ab1-0ddd-474c-95ea-a511562f4f74 | available | volC | 2 | tripleo | false | | | 67c271f4-a180-42f5-8c2f-8a06781668e0 | available | volA | 2 | tripleo | false | | | 8355b998-714c-474d-bd5a-2581eaf21ef8 | available | volE | 2 | tripleo | false | | | be05e68c-a2b9-4f5f-a835-f563e1a89138 | available | volD | 2 | tripleo | false | | | f1d503f8-9398-4f4a-9a3e-cee4e7c431f9 | available | volB | 2 | tripleo | false | | +--------------------------------------+-----------+------+------+-------------+----------+-------------+ All this was done with the default of [root@controller-0 ~]# grep -irn rbd_max /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf 2990:#rbd_max_clone_depth = 5 Now lets kick it down to 1 and restart c-vol rbd_max_clone_depth = 1 And check c-vol log to see if we hit any error, nothing yet. Lets clone another volume again after setting max depth to 1 (overcloud) [stack@undercloud-0 ~]$ cinder create 2 --source-volid 07aa9608-64d0-43f6-a232-da6524de6309 --name volH +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2021-03-31T10:46:09.000000 | | description | None | | encrypted | False | | id | 7e5d9fda-f406-42d0-8f3d-b5c506524e16 | | metadata | {} | | migration_status | None | | multiattach | False | | name | volH | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 7e21ac2b38824e6a9b4c686baed65772 | | replication_status | None | | size | 2 | | snapshot_id | None | | source_volid | 07aa9608-64d0-43f6-a232-da6524de6309 | | status | creating | | updated_at | None | | user_id | b000f0a966e84c14affe64e43b37a2d8 | | volume_type | tripleo | +--------------------------------+--------------------------------------+ All volumes OK: (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------+------+-------------+----------+-------------+ | 07aa9608-64d0-43f6-a232-da6524de6309 | available | volG | 2 | tripleo | false | | | 1cfa6578-ba38-48d1-bbf0-149b07bd6c9c | available | volF | 2 | tripleo | false | | | 1f2b5ab1-0ddd-474c-95ea-a511562f4f74 | available | volC | 2 | tripleo | false | | | 67c271f4-a180-42f5-8c2f-8a06781668e0 | available | volA | 2 | tripleo | false | | | 7e5d9fda-f406-42d0-8f3d-b5c506524e16 | available | volH | 2 | tripleo | false | | | 8355b998-714c-474d-bd5a-2581eaf21ef8 | available | volE | 2 | tripleo | false | | | be05e68c-a2b9-4f5f-a835-f563e1a89138 | available | volD | 2 | tripleo | false | | | f1d503f8-9398-4f4a-9a3e-cee4e7c431f9 | available | volB | 2 | tripleo | false | | +--------------------------------------+-----------+------+------+-------------+----------+-------------+ On c-vol log we see the last two cases where max depth was till 5 and after it was changed to 1: _defaults.rbd_max_clone_depth = 5 log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2589 1101:2021-03-31 10:39:32.046 46 INFO cinder.volume.drivers.rbd [req-4a06ebcc-5a34-4e74-bc47-02d5c96414eb b000f0a966e84c14affe64e43b37a2d8 7e21ac2b38824e6a9b4c686baed65772 - default default] maximum clone depth (5) has been reached - flattening dest volume 1519:2021-03-31 10:45:29.102 8 DEBUG oslo_service.service [req-e548242d-e416-4730-a67a-fefc36d0aed0 - - - - -] tripleo_ceph.rbd_max_clone_depth = None log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2589 1662:2021-03-31 10:45:29.133 8 DEBUG oslo_service.service [req-e548242d-e416-4730-a67a-fefc36d0aed0 - - - - -] backend_defaults.rbd_max_clone_depth = 1 log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2589 Another test lets create a new volume from an image: (overcloud) [stack@undercloud-0 ~]$ cinder create 1 --name Newvol --image 680ef9d5-e7ec-4f6d-9927-e0ba636fe4eb +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2021-03-31T10:49:00.000000 | | description | None | | encrypted | False | | id | b6943cb3-2873-4af3-b762-7db4c791a85f | | metadata | {} | | migration_status | None | | multiattach | False | | name | Newvol | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 7e21ac2b38824e6a9b4c686baed65772 | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | updated_at | None | | user_id | b000f0a966e84c14affe64e43b37a2d8 | | volume_type | tripleo | +--------------------------------+--------------------------------------+ And now lets clone this new volume: (overcloud) [stack@undercloud-0 ~]$ cinder create 2 --source-volid b6943cb3-2873-4af3-b762-7db4c791a85f --name NewCloned +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | true | | consistencygroup_id | None | | created_at | 2021-03-31T10:49:55.000000 | | description | None | | encrypted | False | | id | dea3fae7-27a1-4f52-ad7e-a632f8332efa | | metadata | {} | | migration_status | None | | multiattach | False | | name | NewCloned | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 7e21ac2b38824e6a9b4c686baed65772 | | replication_status | None | | size | 2 | | snapshot_id | None | | source_volid | b6943cb3-2873-4af3-b762-7db4c791a85f | | status | creating | | updated_at | None | | user_id | b000f0a966e84c14affe64e43b37a2d8 | | volume_type | tripleo | +--------------------------------+--------------------------------------+ Again now errors on c-vol log all volumes are avaliable and we notice the depth reached for 1 too: _ceph.rbd_max_clone_depth = None log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2589 516:2021-03-31 09:35:39.681 7 DEBUG oslo_service.service [req-4e959c96-362b-413b-9dfa-f08104d3726f - - - - -] backend_defaults.rbd_max_clone_depth = 5 log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2589 1101:2021-03-31 10:39:32.046 46 INFO cinder.volume.drivers.rbd [req-4a06ebcc-5a34-4e74-bc47-02d5c96414eb b000f0a966e84c14affe64e43b37a2d8 7e21ac2b38824e6a9b4c686baed65772 - default default] maximum clone depth (5) has been reached - flattening dest volume 1519:2021-03-31 10:45:29.102 8 DEBUG oslo_service.service [req-e548242d-e416-4730-a67a-fefc36d0aed0 - - - - -] tripleo_ceph.rbd_max_clone_depth = None log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2589 1662:2021-03-31 10:45:29.133 8 DEBUG oslo_service.service [req-e548242d-e416-4730-a67a-fefc36d0aed0 - - - - -] backend_defaults.rbd_max_clone_depth = 1 log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2589 1905:2021-03-31 10:53:42.099 26 INFO cinder.volume.drivers.rbd [req-5c965ea0-1875-4eb6-a8c5-0351c6e7ba82 b000f0a966e84c14affe64e43b37a2d8 7e21ac2b38824e6a9b4c686baed65772 - default default] maximum clone depth (1) has been reached - flattening dest volume Good to verify.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:3483