Bug 2005966
| Summary: | [RHOSP16.1] Image upload for qcow file failed on NFS cinder backend | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Luigi Tamagnone <ltamagno> |
| Component: | python-glance-store | Assignee: | Rajat Dhasmana <rdhasman> |
| Status: | CLOSED ERRATA | QA Contact: | Tzach Shefi <tshefi> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 16.1 (Train) | CC: | apevec, cyril, gcharot, gfidente, lhh, pambre, rdhasman, tshefi |
| Target Milestone: | z8 | Keywords: | Triaged |
| Target Release: | 16.1 (Train on RHEL 8.2) | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | python-glance-store-1.0.2-1.20220110223455.bc62bb4 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-03-24 11:01:35 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 2009039 | ||
| Bug Blocks: | |||
|
Description
Luigi Tamagnone
2021-09-20 14:56:00 UTC
Hello, Looking at the same log file as you (0140-03031713_sosreport-control-3-03031713-2021-09-20-frnkqfu.tar.xz/sosreport-control-3-03031713-2021-09-20-frnkqfu/var/log/containers/glance/api.log.1), I can find this: Invalid volume: Volume status must be available to reserve, but the status is attaching. (HTTP 400) (Request-ID: req-98559631-3768-493c-af33-23bd06433a97): cinderclient.exceptions.BadRequest: Invalid volume: Volume status must be available to reserve, but the status is attaching. (HTTP 400) (Request-ID: req-98559631-3768-493c-af33-23bd06433a97) This raises a BackendException which in turn causes glance_store to raise a StorageFull exception, thus leading to the output you see. I think the only fix is to implement multiattach here. @Rajat: Can you confirm that what I wrote above is right? Verified on:
python3-glance-store-1.0.2-1.20220110223456.el8ost.noarch
On a deployment where Glance uses Cinder, which in turn uses a Netapp NFS backend (vers=4.1)
I've uploaded a cirros qcow2 image:
(overcloud) [stack@undercloud-0 ~]$ glance image-show 7fefdb47-1a6c-4d73-883e-abfc45f22a73
+----------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+----------------------------------+----------------------------------------------------------------------------------+
| checksum | 443b7623e27ecf03dc9e01ee93f67afe |
| container_format | bare |
| created_at | 2022-03-03T10:44:00Z |
| direct_url | cinder://59f1998d-645e-4c4a-8935-908730773b53 |
| disk_format | qcow2 |
| id | 7fefdb47-1a6c-4d73-883e-abfc45f22a73 |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros |
| os_hash_algo | sha512 |
| os_hash_value | 6513f21e44aa3da349f248188a44bc304a3653a04122d8fb4535423c8e1d14cd6a153f735bb0982e |
| | 2161b5b5186106570c17a9e58b64dd39390617cd5a350f78 |
| os_hidden | False |
| owner | 8cd92774d2a24296b519b8f6382781c6 |
| owner_specified.openstack.md5 | |
| owner_specified.openstack.object | images/cirros |
| owner_specified.openstack.sha256 | |
| protected | False |
| size | 12716032 |
| status | active |
| stores | default_backend |
| tags | [] |
| updated_at | 2022-03-03T10:44:27Z |
| virtual_size | Not available |
| visibility | public |
+----------------------------------+----------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ cinder list --all
+--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+
| ID | Tenant ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+
| 59f1998d-645e-4c4a-8935-908730773b53 | 2332dec9e9d34e51bc0a9e7e57930439 | available | image-7fefdb47-1a6c-4d73-883e-abfc45f22a73 | 1 | tripleo | false | |
+--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+
Now lets try a few more images
First CentOS-8-GenericCloud-8.1.1911-20200113.3.x86_64.qcow2
(overcloud) [stack@undercloud-0 ~]$ qemu-img info CentOS-8-GenericCloud-8.1.1911-20200113.3.x86_64.qcow2
image: CentOS-8-GenericCloud-8.1.1911-20200113.3.x86_64.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 683 MiB
cluster_size: 65536
Format specific information:
compat: 0.10
refcount bits: 16
(overcloud) [stack@undercloud-0 ~]$ openstack image create --disk-format qcow2 --file CentOS-8-GenericCloud-8.1.1911-20200113.3.x86_64.qcow2 --container-format bare CentOS_683MiB
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum | d89eb49f2c264d29225cecf2b6c83322 |
| container_format | bare |
| created_at | 2022-03-07T14:27:49Z |
| disk_format | qcow2 |
| file | /v2/images/c72a5030-1a0b-42a6-9e44-f4b8c58f304e/file |
| id | c72a5030-1a0b-42a6-9e44-f4b8c58f304e |
| min_disk | 0 |
| min_ram | 0 |
| name | CentOS_683MiB |
| owner | 8cd92774d2a24296b519b8f6382781c6 |
| properties | direct_url='cinder://21e7b47a-f26e-4a72-b7ac-db33bd5b14dd', os_hash_algo='sha512', os_hash_value='603dd170927a25bf405cdd60cf6052fef4363ef563e9a7df0d592d370dec3fa79baa77ac37aa39196a2bd27a1b4a5324f5fb12ee7ca7626c8ba90543449760ad', os_hidden='False', stores='default_backend' |
| protected | False |
| schema | /v2/schemas/image |
| size | 716176896 |
| status | active |
| tags | |
| updated_at | 2022-03-07T14:28:28Z |
| virtual_size | None |
| visibility | shared |
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ glance image-show c72a5030-1a0b-42a6-9e44-f4b8c58f304e
+------------------+----------------------------------------------------------------------------------+
| Property | Value |
+------------------+----------------------------------------------------------------------------------+
| checksum | d89eb49f2c264d29225cecf2b6c83322 |
| container_format | bare |
| created_at | 2022-03-07T14:27:49Z |
| direct_url | cinder://21e7b47a-f26e-4a72-b7ac-db33bd5b14dd |
| disk_format | qcow2 |
| id | c72a5030-1a0b-42a6-9e44-f4b8c58f304e |
| min_disk | 0 |
| min_ram | 0 |
| name | CentOS_683MiB |
| os_hash_algo | sha512 |
| os_hash_value | 603dd170927a25bf405cdd60cf6052fef4363ef563e9a7df0d592d370dec3fa79baa77ac37aa3919 |
| | 6a2bd27a1b4a5324f5fb12ee7ca7626c8ba90543449760ad |
| os_hidden | False |
| owner | 8cd92774d2a24296b519b8f6382781c6 |
| protected | False |
| size | 716176896 |
| status | active |
| stores | default_backend |
| tags | [] |
| updated_at | 2022-03-07T14:28:28Z |
| virtual_size | Not available |
| visibility | shared |
+------------------+----------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack image list
+--------------------------------------+---------------+--------+
| ID | Name | Status |
+--------------------------------------+---------------+--------+
| c72a5030-1a0b-42a6-9e44-f4b8c58f304e | CentOS_683MiB | active |
| 7fefdb47-1a6c-4d73-883e-abfc45f22a73 | cirros | active |
+--------------------------------------+---------------+--------+
(overcloud) [stack@undercloud-0 ~]$ cinder list --all
+--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+
| ID | Tenant ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+
| 21e7b47a-f26e-4a72-b7ac-db33bd5b14dd | 2332dec9e9d34e51bc0a9e7e57930439 | available | image-c72a5030-1a0b-42a6-9e44-f4b8c58f304e | 1 | tripleo | false | |
| 59f1998d-645e-4c4a-8935-908730773b53 | 2332dec9e9d34e51bc0a9e7e57930439 | available | image-7fefdb47-1a6c-4d73-883e-abfc45f22a73 | 1 | tripleo | false | |
+--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+
Again a qcow2 image was successfully uploaded to Glance over Cinder over nfs.
And once last time again a Centos image but larger than 1G this time
(overcloud) [stack@undercloud-0 ~]$ qemu-img info CentOS-8-GenericCloud-8.4.2105-20210603.0.x86_64.qcow2
image: CentOS-8-GenericCloud-8.4.2105-20210603.0.x86_64.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 1.22 GiB
cluster_size: 65536
Format specific information:
compat: 0.10
refcount bits: 16
(overcloud) [stack@undercloud-0 ~]$ openstack image create --disk-format qcow2 --file CentOS-8-GenericCloud-8.4.2105-20210603.0.x86_64.qcow2 --container-format bare CentOS_1.22GiB
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum | 032eed270415526546eac07628905a62 |
| container_format | bare |
| created_at | 2022-03-07T14:32:01Z |
| disk_format | qcow2 |
| file | /v2/images/6f7a5686-69fd-42c9-86ad-672ea45def6d/file |
| id | 6f7a5686-69fd-42c9-86ad-672ea45def6d |
| min_disk | 0 |
| min_ram | 0 |
| name | CentOS_1.22GiB |
| owner | 8cd92774d2a24296b519b8f6382781c6 |
| properties | direct_url='cinder://18adc14b-cafb-4caa-bb9b-3b995ce58202', os_hash_algo='sha512', os_hash_value='bd0df4a58925a5137188b26f646595ccb6ef88feca4dfd3d1c6aa547abc4190ceaf9f75fcd54336b83a77aa4a35d1677264a47ee35a575fbba4e2ef149bcc86b', os_hidden='False', stores='default_backend' |
| protected | False |
| schema | /v2/schemas/image |
| size | 1309652992 |
| status | active |
| tags | |
| updated_at | 2022-03-07T14:32:29Z |
| virtual_size | None |
| visibility | shared |
+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ glance image-show 6f7a5686-69fd-42c9-86ad-672ea45def6d
+------------------+----------------------------------------------------------------------------------+
| Property | Value |
+------------------+----------------------------------------------------------------------------------+
| checksum | 032eed270415526546eac07628905a62 |
| container_format | bare |
| created_at | 2022-03-07T14:32:01Z |
| direct_url | cinder://18adc14b-cafb-4caa-bb9b-3b995ce58202 |
| disk_format | qcow2 |
| id | 6f7a5686-69fd-42c9-86ad-672ea45def6d |
| min_disk | 0 |
| min_ram | 0 |
| name | CentOS_1.22GiB |
| os_hash_algo | sha512 |
| os_hash_value | bd0df4a58925a5137188b26f646595ccb6ef88feca4dfd3d1c6aa547abc4190ceaf9f75fcd54336b |
| | 83a77aa4a35d1677264a47ee35a575fbba4e2ef149bcc86b |
| os_hidden | False |
| owner | 8cd92774d2a24296b519b8f6382781c6 |
| protected | False |
| size | 1309652992 |
| status | active |
| stores | default_backend |
| tags | [] |
| updated_at | 2022-03-07T14:32:29Z |
| virtual_size | Not available |
| visibility | shared |
+------------------+----------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack image list
cinder +--------------------------------------+----------------+--------+
| ID | Name | Status |
+--------------------------------------+----------------+--------+
| 6f7a5686-69fd-42c9-86ad-672ea45def6d | CentOS_1.22GiB | active |
| c72a5030-1a0b-42a6-9e44-f4b8c58f304e | CentOS_683MiB | active |
| 7fefdb47-1a6c-4d73-883e-abfc45f22a73 | cirros | active |
+--------------------------------------+----------------+--------+
(overcloud) [stack@undercloud-0 ~]$ cinder list --all
+--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+
| ID | Tenant ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+
| 18adc14b-cafb-4caa-bb9b-3b995ce58202 | 2332dec9e9d34e51bc0a9e7e57930439 | available | image-6f7a5686-69fd-42c9-86ad-672ea45def6d | 2 | tripleo | false | |
| 21e7b47a-f26e-4a72-b7ac-db33bd5b14dd | 2332dec9e9d34e51bc0a9e7e57930439 | available | image-c72a5030-1a0b-42a6-9e44-f4b8c58f304e | 1 | tripleo | false | |
| 59f1998d-645e-4c4a-8935-908730773b53 | 2332dec9e9d34e51bc0a9e7e57930439 | available | image-7fefdb47-1a6c-4d73-883e-abfc45f22a73 | 1 | tripleo | false | |
+--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+
All three qcow2 images successfully uploaded to Glance over Cinder over netapp NFS v4.1, good to verify.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.8 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:0986 |