Bug 1925312 - RBD Driver: Ceph Pacific support
Summary: RBD Driver: Ceph Pacific support
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 17.0 (Wallaby)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: Alpha
: 17.0
Assignee: Jon Bernard
QA Contact: Tzach Shefi
RHOS Documentation Team
URL:
Whiteboard:
Depends On: 1923789
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-02-04 20:27 UTC by Eric Harney
Modified: 2022-09-21 12:14 UTC (History)
12 users (show)

Fixed In Version: openstack-cinder-18.2.1-0.20220605050357.9a473fd.el9ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-09-21 12:13:29 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1931004 0 None None None 2021-06-06 10:35:12 UTC
OpenStack gerrit 786260 0 None master: MERGED cinder: RBD: use correct stripe unit in clone operation (Iec111ab83e9ed8182c9679c911e3d90927d5a7c3) 2022-06-28 14:34:47 UTC
OpenStack gerrit 804265 0 None stable/wallaby: MERGED cinder: RBD: use correct stripe unit in clone operation (Iec111ab83e9ed8182c9679c911e3d90927d5a7c3) 2022-06-28 14:34:51 UTC
Red Hat Issue Tracker OSP-1739 0 None None None 2022-06-13 18:52:10 UTC
Red Hat Product Errata RHEA-2022:6543 0 None None None 2022-09-21 12:14:33 UTC

Internal Links: 2002772

Description Eric Harney 2021-02-04 20:27:36 UTC
Description of problem:
Ensure that the RBD driver supports RBD Pacific.

Version-Release number of selected component (if applicable):
OSP17

Comment 4 Paul Grist 2021-02-18 16:23:50 UTC
Bumping up the priority as we need to have Ceph Pacific working with the RBD driver before M3 closes

Comment 10 Jon Bernard 2021-04-14 15:52:22 UTC
Here is what's happening:

In Pacific a check was added to make sure during a clone operation that the child's strip unit was not less than that of its parent.  Failing this condition returns -EINVAL, which is then raised by python-rbd as an exception.  This maps to the 'order' argument in clone(), where order is log base 2 of the strip unit.  Ceph's default is 4 megabytes.  The reason we're seeing EINVAL exceptions in the Pacific CI is that: when Openstack is configured to use Ceph for both cinder and glance, volume-from-image tests fail because Glance's default stripe unit is 8 (distinctly larger than Cinder's 4). This results in an order calculation of 22, which is invalid for clone() (too small).

I see two possible solutions and have proposed patches linked to this BZ for both:

1. Increase Cinder's default chunk size to match Glance's.  I think this makes sense for both consistency and performance.

2. When doing a clone(), consider the configured chunk size /and/ the strip unit of the parent volume and choose the higher value.

Either of these approaches prevent the failures we're seeing, I think they are both useful individually as well.

Comment 17 Tzach Shefi 2022-07-03 08:32:48 UTC
Hi Jon(or Eric), 

Per verification, 
On a Ceph backed deployment, upload a RAW image to Glance,
Create a volume from that image, volume should be available.

Is there anything else I should test for? 
Thanks

Comment 18 Jon Bernard 2022-07-06 11:03:00 UTC
That should do it.  Previously, volume-from-image with a glance image in raw form would allow cinder to clone the image, resulting in failure.  If that operation (volume-from-raw-image) succeeds for pacific and above, this patch should be working.

Comment 19 Tzach Shefi 2022-07-06 11:42:51 UTC
Verified on:
openstack-cinder-18.2.1-0.20220605050357.9a473fd.el9ost.noarch

On a Ceph backed deployment, Lets upload a few raw images and create volumes from them.


First image:

[stack@undercloud-0 ~]$ glance image-create --disk-format raw --container-format bare --file CentOS-Stream-GenericCloud-9-20220606.0.x86_64.raw --name CentOS-Stream-GenericCloud-9 --progress
[=============================>] 100%
+------------------+----------------------------------------------------------------------------------+
| Property         | Value                                                                            |
+------------------+----------------------------------------------------------------------------------+
| checksum         | f6410d5fb422ebf152d1a8876deb21c7                                                 |
| container_format | bare                                                                             |
| created_at       | 2022-07-06T11:22:36Z                                                             |
| direct_url       | rbd://fd7ad824-a913-5b2f-acfc-                                                   |
|                  | fabb670e0ebc/images/8f191dc7-e0ba-4393-bc11-5db8dd83a423/snap                    |
| disk_format      | raw                                                                              |
| id               | 8f191dc7-e0ba-4393-bc11-5db8dd83a423                                             |
| locations        | [{"url": "rbd://fd7ad824-a913-5b2f-acfc-                                         |
|                  | fabb670e0ebc/images/8f191dc7-e0ba-4393-bc11-5db8dd83a423/snap", "metadata":      |
|                  | {"store": "default_backend"}}]                                                   |
| min_disk         | 0                                                                                |
| min_ram          | 0                                                                                |
| name             | CentOS-Stream-GenericCloud-9                                                     |
| os_hash_algo     | sha512                                                                           |
| os_hash_value    | adc0d6cb4e000d0a1333f22b24a65a304769164358fe794c860abeafbe1336ac9c509a6cd83ddde1 |
|                  | d10cf6117a593160d4a7e95778e2a2819fbe66c35a499eaf                                 |
| os_hidden        | False                                                                            |
| owner            | 78cfade4d5854083964e41ac71921565                                                 |
| protected        | False                                                                            |
| size             | 10737418240                                                                      |
| status           | active                                                                           |
| stores           | default_backend                                                                  |
| tags             | []                                                                               |
| updated_at       | 2022-07-06T11:24:43Z                                                             |
| virtual_size     | 10737418240                                                                      |
| visibility       | shared                                                                           |
+------------------+----------------------------------------------------------------------------------+

Active, raw backed by Ceph, lets create a volume from it:
[stack@undercloud-0 ~]$ cinder create 10 --image CentOS-Stream-GenericCloud-9 --name VolFromCentOSImage
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2022-07-06T11:26:36.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 631e065c-75ad-4433-8b82-ffc885547096 |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | VolFromCentOSImage                   |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 78cfade4d5854083964e41ac71921565     |
| replication_status             | None                                 |
| size                           | 10                                   |
| snapshot_id                    | None                                 |
| source_volid                   | None                                 |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | 85321f14fc344d228df218d103c178a8     |
| volume_type                    | tripleo_default                      |
+--------------------------------+--------------------------------------+


 [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+----------------------+------+-----------------+----------+-------------+
| ID                                   | Status    | Name                 | Size | Volume Type     | Bootable | Attached to |
+--------------------------------------+-----------+----------------------+------+-----------------+----------+-------------+
| 631e065c-75ad-4433-8b82-ffc885547096 | available | VolFromCentOSImage   | 10   | tripleo_default | true     |             |


Looks good, just to be sure lets test again, with another image:
[stack@undercloud-0 ~]$ glance image-create --disk-format raw --container-format bare --file rhel-server-7.9-update-12-x86_64-kvm.raw --name rhel-server-7.9 --progress
[=============================>] 100%
+------------------+----------------------------------------------------------------------------------+
| Property         | Value                                                                            |
+------------------+----------------------------------------------------------------------------------+
| checksum         | 115061285377b8bb9061fa165954f764                                                 |
| container_format | bare                                                                             |
| created_at       | 2022-07-06T11:30:13Z                                                             |
| direct_url       | rbd://fd7ad824-a913-5b2f-acfc-                                                   |
|                  | fabb670e0ebc/images/a67fb5ff-7c22-4cb6-b8b9-b4e5571b0f2d/snap                    |
| disk_format      | raw                                                                              |
| id               | a67fb5ff-7c22-4cb6-b8b9-b4e5571b0f2d                                             |
| locations        | [{"url": "rbd://fd7ad824-a913-5b2f-acfc-                                         |
|                  | fabb670e0ebc/images/a67fb5ff-7c22-4cb6-b8b9-b4e5571b0f2d/snap", "metadata":      |
|                  | {"store": "default_backend"}}]                                                   |
| min_disk         | 0                                                                                |
| min_ram          | 0                                                                                |
| name             | rhel-server-7.9                                                                  |
| os_hash_algo     | sha512                                                                           |
| os_hash_value    | 66bd8bef4a6b0cfae1ca3e87f1e2dd6e9d55768a101a451970a9e44cb66acf92c2d2e46431866423 |
|                  | 2f852a433ce5b4287d1027427e79ef1a17e54f7a8fdfc675                                 |
| os_hidden        | False                                                                            |
| owner            | 78cfade4d5854083964e41ac71921565                                                 |
| protected        | False                                                                            |
| size             | 10737418240                                                                      |
| status           | active                                                                           |
| stores           | default_backend                                                                  |
| tags             | []                                                                               |
| updated_at       | 2022-07-06T11:32:21Z                                                             |
| virtual_size     | 10737418240                                                                      |
| visibility       | shared                                                                           |
+------------------+----------------------------------------------------------------------------------+

Now a volume from this new image:
[stack@undercloud-0 ~]$  cinder create 10 --image rhel-server-7.9 --name VolFromImagerhel-server-7.9
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2022-07-06T11:39:48.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 68badeaa-98e5-4cf8-9ded-24a3af4b0bb2 |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | VolFromImagerhel-server-7.9          |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 78cfade4d5854083964e41ac71921565     |
| replication_status             | None                                 |
| size                           | 10                                   |
| snapshot_id                    | None                                 |
| source_volid                   | None                                 |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | 85321f14fc344d228df218d103c178a8     |
| volume_type                    | tripleo_default                      |
+--------------------------------+--------------------------------------+

Yep, this second volume is also available.  
 [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+-----------------------------+------+-----------------+----------+-------------+
| ID                                   | Status    | Name                        | Size | Volume Type     | Bootable | Attached to |
+--------------------------------------+-----------+-----------------------------+------+-----------------+----------+-------------+
| 631e065c-75ad-4433-8b82-ffc885547096 | available | VolFromCentOSImage          | 10   | tripleo_default | true     |             |
| 68badeaa-98e5-4cf8-9ded-24a3af4b0bb2 | available | VolFromImagerhel-server-7.9 | 10   | tripleo_default | true     |             |



Good to verify, we successfully created two volumes one from each image of the raw images.

Comment 25 errata-xmlrpc 2022-09-21 12:13:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543


Note You need to log in before you can comment on or make changes to this bug.