Description of problem: Trying to create a new image with a qcow file failed[1]. This issue happens in an env where Cinder is the backend of Glance and NetApp NFSv4 is the backend of Cinder. It could be bind with BUG 1838653, but workaround[2] doesn't work. Version-Release number of selected component (if applicable): [os-release] Red Hat Enterprise Linux 8.2 (Ootpa) 8.2 (Ootpa) [rhosp-release] Red Hat OpenStack Platform release 16.1.6 GA (Train) How reproducible: The issue is easy to reproduce on all Steps to Reproduce: 1. openstack image create --disk-format qcow2 --file CentOS-8.qcow2 CentOS-8-test22 --container-format bare Actual results: Image creation failed Expected results: image creation should succeed Additional info: [1] It can fail with two outputs in a period way 1 time: glanceclient.exc.HTTPInternalServerError: HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. and two times: glanceclient.exc.HTTPOverLimit: HTTP 413 Request Entity Too Large: Image storage media is full: There is not enough disk space on the image storage media. [2] https://access.redhat.com/solutions/5499481 qemu-img convert -S 0 -f qcow2 -O qcow2 CentOS-8.qcow2 new.qcow2
Hello, Looking at the same log file as you (0140-03031713_sosreport-control-3-03031713-2021-09-20-frnkqfu.tar.xz/sosreport-control-3-03031713-2021-09-20-frnkqfu/var/log/containers/glance/api.log.1), I can find this: Invalid volume: Volume status must be available to reserve, but the status is attaching. (HTTP 400) (Request-ID: req-98559631-3768-493c-af33-23bd06433a97): cinderclient.exceptions.BadRequest: Invalid volume: Volume status must be available to reserve, but the status is attaching. (HTTP 400) (Request-ID: req-98559631-3768-493c-af33-23bd06433a97) This raises a BackendException which in turn causes glance_store to raise a StorageFull exception, thus leading to the output you see. I think the only fix is to implement multiattach here. @Rajat: Can you confirm that what I wrote above is right?
Verified on: python3-glance-store-1.0.2-1.20220110223456.el8ost.noarch On a deployment where Glance uses Cinder, which in turn uses a Netapp NFS backend (vers=4.1) I've uploaded a cirros qcow2 image: (overcloud) [stack@undercloud-0 ~]$ glance image-show 7fefdb47-1a6c-4d73-883e-abfc45f22a73 +----------------------------------+----------------------------------------------------------------------------------+ | Property | Value | +----------------------------------+----------------------------------------------------------------------------------+ | checksum | 443b7623e27ecf03dc9e01ee93f67afe | | container_format | bare | | created_at | 2022-03-03T10:44:00Z | | direct_url | cinder://59f1998d-645e-4c4a-8935-908730773b53 | | disk_format | qcow2 | | id | 7fefdb47-1a6c-4d73-883e-abfc45f22a73 | | min_disk | 0 | | min_ram | 0 | | name | cirros | | os_hash_algo | sha512 | | os_hash_value | 6513f21e44aa3da349f248188a44bc304a3653a04122d8fb4535423c8e1d14cd6a153f735bb0982e | | | 2161b5b5186106570c17a9e58b64dd39390617cd5a350f78 | | os_hidden | False | | owner | 8cd92774d2a24296b519b8f6382781c6 | | owner_specified.openstack.md5 | | | owner_specified.openstack.object | images/cirros | | owner_specified.openstack.sha256 | | | protected | False | | size | 12716032 | | status | active | | stores | default_backend | | tags | [] | | updated_at | 2022-03-03T10:44:27Z | | virtual_size | Not available | | visibility | public | +----------------------------------+----------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ cinder list --all +--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+ | ID | Tenant ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+ | 59f1998d-645e-4c4a-8935-908730773b53 | 2332dec9e9d34e51bc0a9e7e57930439 | available | image-7fefdb47-1a6c-4d73-883e-abfc45f22a73 | 1 | tripleo | false | | +--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+ Now lets try a few more images First CentOS-8-GenericCloud-8.1.1911-20200113.3.x86_64.qcow2 (overcloud) [stack@undercloud-0 ~]$ qemu-img info CentOS-8-GenericCloud-8.1.1911-20200113.3.x86_64.qcow2 image: CentOS-8-GenericCloud-8.1.1911-20200113.3.x86_64.qcow2 file format: qcow2 virtual size: 10 GiB (10737418240 bytes) disk size: 683 MiB cluster_size: 65536 Format specific information: compat: 0.10 refcount bits: 16 (overcloud) [stack@undercloud-0 ~]$ openstack image create --disk-format qcow2 --file CentOS-8-GenericCloud-8.1.1911-20200113.3.x86_64.qcow2 --container-format bare CentOS_683MiB +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | checksum | d89eb49f2c264d29225cecf2b6c83322 | | container_format | bare | | created_at | 2022-03-07T14:27:49Z | | disk_format | qcow2 | | file | /v2/images/c72a5030-1a0b-42a6-9e44-f4b8c58f304e/file | | id | c72a5030-1a0b-42a6-9e44-f4b8c58f304e | | min_disk | 0 | | min_ram | 0 | | name | CentOS_683MiB | | owner | 8cd92774d2a24296b519b8f6382781c6 | | properties | direct_url='cinder://21e7b47a-f26e-4a72-b7ac-db33bd5b14dd', os_hash_algo='sha512', os_hash_value='603dd170927a25bf405cdd60cf6052fef4363ef563e9a7df0d592d370dec3fa79baa77ac37aa39196a2bd27a1b4a5324f5fb12ee7ca7626c8ba90543449760ad', os_hidden='False', stores='default_backend' | | protected | False | | schema | /v2/schemas/image | | size | 716176896 | | status | active | | tags | | | updated_at | 2022-03-07T14:28:28Z | | virtual_size | None | | visibility | shared | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ glance image-show c72a5030-1a0b-42a6-9e44-f4b8c58f304e +------------------+----------------------------------------------------------------------------------+ | Property | Value | +------------------+----------------------------------------------------------------------------------+ | checksum | d89eb49f2c264d29225cecf2b6c83322 | | container_format | bare | | created_at | 2022-03-07T14:27:49Z | | direct_url | cinder://21e7b47a-f26e-4a72-b7ac-db33bd5b14dd | | disk_format | qcow2 | | id | c72a5030-1a0b-42a6-9e44-f4b8c58f304e | | min_disk | 0 | | min_ram | 0 | | name | CentOS_683MiB | | os_hash_algo | sha512 | | os_hash_value | 603dd170927a25bf405cdd60cf6052fef4363ef563e9a7df0d592d370dec3fa79baa77ac37aa3919 | | | 6a2bd27a1b4a5324f5fb12ee7ca7626c8ba90543449760ad | | os_hidden | False | | owner | 8cd92774d2a24296b519b8f6382781c6 | | protected | False | | size | 716176896 | | status | active | | stores | default_backend | | tags | [] | | updated_at | 2022-03-07T14:28:28Z | | virtual_size | Not available | | visibility | shared | +------------------+----------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack image list +--------------------------------------+---------------+--------+ | ID | Name | Status | +--------------------------------------+---------------+--------+ | c72a5030-1a0b-42a6-9e44-f4b8c58f304e | CentOS_683MiB | active | | 7fefdb47-1a6c-4d73-883e-abfc45f22a73 | cirros | active | +--------------------------------------+---------------+--------+ (overcloud) [stack@undercloud-0 ~]$ cinder list --all +--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+ | ID | Tenant ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+ | 21e7b47a-f26e-4a72-b7ac-db33bd5b14dd | 2332dec9e9d34e51bc0a9e7e57930439 | available | image-c72a5030-1a0b-42a6-9e44-f4b8c58f304e | 1 | tripleo | false | | | 59f1998d-645e-4c4a-8935-908730773b53 | 2332dec9e9d34e51bc0a9e7e57930439 | available | image-7fefdb47-1a6c-4d73-883e-abfc45f22a73 | 1 | tripleo | false | | +--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+ Again a qcow2 image was successfully uploaded to Glance over Cinder over nfs. And once last time again a Centos image but larger than 1G this time (overcloud) [stack@undercloud-0 ~]$ qemu-img info CentOS-8-GenericCloud-8.4.2105-20210603.0.x86_64.qcow2 image: CentOS-8-GenericCloud-8.4.2105-20210603.0.x86_64.qcow2 file format: qcow2 virtual size: 10 GiB (10737418240 bytes) disk size: 1.22 GiB cluster_size: 65536 Format specific information: compat: 0.10 refcount bits: 16 (overcloud) [stack@undercloud-0 ~]$ openstack image create --disk-format qcow2 --file CentOS-8-GenericCloud-8.4.2105-20210603.0.x86_64.qcow2 --container-format bare CentOS_1.22GiB +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | checksum | 032eed270415526546eac07628905a62 | | container_format | bare | | created_at | 2022-03-07T14:32:01Z | | disk_format | qcow2 | | file | /v2/images/6f7a5686-69fd-42c9-86ad-672ea45def6d/file | | id | 6f7a5686-69fd-42c9-86ad-672ea45def6d | | min_disk | 0 | | min_ram | 0 | | name | CentOS_1.22GiB | | owner | 8cd92774d2a24296b519b8f6382781c6 | | properties | direct_url='cinder://18adc14b-cafb-4caa-bb9b-3b995ce58202', os_hash_algo='sha512', os_hash_value='bd0df4a58925a5137188b26f646595ccb6ef88feca4dfd3d1c6aa547abc4190ceaf9f75fcd54336b83a77aa4a35d1677264a47ee35a575fbba4e2ef149bcc86b', os_hidden='False', stores='default_backend' | | protected | False | | schema | /v2/schemas/image | | size | 1309652992 | | status | active | | tags | | | updated_at | 2022-03-07T14:32:29Z | | virtual_size | None | | visibility | shared | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ glance image-show 6f7a5686-69fd-42c9-86ad-672ea45def6d +------------------+----------------------------------------------------------------------------------+ | Property | Value | +------------------+----------------------------------------------------------------------------------+ | checksum | 032eed270415526546eac07628905a62 | | container_format | bare | | created_at | 2022-03-07T14:32:01Z | | direct_url | cinder://18adc14b-cafb-4caa-bb9b-3b995ce58202 | | disk_format | qcow2 | | id | 6f7a5686-69fd-42c9-86ad-672ea45def6d | | min_disk | 0 | | min_ram | 0 | | name | CentOS_1.22GiB | | os_hash_algo | sha512 | | os_hash_value | bd0df4a58925a5137188b26f646595ccb6ef88feca4dfd3d1c6aa547abc4190ceaf9f75fcd54336b | | | 83a77aa4a35d1677264a47ee35a575fbba4e2ef149bcc86b | | os_hidden | False | | owner | 8cd92774d2a24296b519b8f6382781c6 | | protected | False | | size | 1309652992 | | status | active | | stores | default_backend | | tags | [] | | updated_at | 2022-03-07T14:32:29Z | | virtual_size | Not available | | visibility | shared | +------------------+----------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack image list cinder +--------------------------------------+----------------+--------+ | ID | Name | Status | +--------------------------------------+----------------+--------+ | 6f7a5686-69fd-42c9-86ad-672ea45def6d | CentOS_1.22GiB | active | | c72a5030-1a0b-42a6-9e44-f4b8c58f304e | CentOS_683MiB | active | | 7fefdb47-1a6c-4d73-883e-abfc45f22a73 | cirros | active | +--------------------------------------+----------------+--------+ (overcloud) [stack@undercloud-0 ~]$ cinder list --all +--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+ | ID | Tenant ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+ | 18adc14b-cafb-4caa-bb9b-3b995ce58202 | 2332dec9e9d34e51bc0a9e7e57930439 | available | image-6f7a5686-69fd-42c9-86ad-672ea45def6d | 2 | tripleo | false | | | 21e7b47a-f26e-4a72-b7ac-db33bd5b14dd | 2332dec9e9d34e51bc0a9e7e57930439 | available | image-c72a5030-1a0b-42a6-9e44-f4b8c58f304e | 1 | tripleo | false | | | 59f1998d-645e-4c4a-8935-908730773b53 | 2332dec9e9d34e51bc0a9e7e57930439 | available | image-7fefdb47-1a6c-4d73-883e-abfc45f22a73 | 1 | tripleo | false | | +--------------------------------------+----------------------------------+-----------+--------------------------------------------+------+-------------+----------+-------------+ All three qcow2 images successfully uploaded to Glance over Cinder over netapp NFS v4.1, good to verify.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.8 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:0986