Description of problem: Enable new configuration option ``enable_thin_provisioning`` in OOO, to rbd and filesystem store to enable or not sparse upload. Sparse file mean not really write null byte sequences but only the data itself at a given offset, the "holes" who can appear will automatically interpreted by the storage backend as null bytes, and do not really consume your storage. BP : https://blueprints.launchpad.net/glance-store/+spec/handle-sparse-image
https://review.opendev.org/#/c/744282/9/releasenotes/notes/handle-sparse-image-a3ecfc4ae1c00d48.yaml
Actually, the RPMs have been created: python-glance-store-1.0.2-2.20210205164924.bc62bb4.el8 openstack-tripleo-heat-templates-11.4.1-2.20210208205916.dfc4da5.el8 puppet-tripleo-11.5.1-2.20210209011041.0d42695.el8 puppet-glance-15.4.1-2.20210208204921.aa0e036.el8 I'm updating the "Fixed-in" version and moving this to MODIFIED.
fixed-in versions present: [root@controller-0 /]# rpm -qa python3-glance-store openstack-tripleo-heat-templates puppet-tripleo puppet-glance python3-glance-store-1.0.2-2.20210225105026.79e043a.el8ost.1.noarch [root@controller-0 /]# exit [root@controller-0 ~]# logout [heat-admin@controller-0 ~]$ rpm -qa python3-glance-store openstack-tripleo-heat-templates puppet-tripleo puppet-glance puppet-glance-15.4.1-2.20210324004946.4b4b990.el8ost.1.noarch puppet-tripleo-11.5.1-2.20210323024955.4d3d23e.el8ost.1.noarch [heat-admin@controller-0 ~]$ logout Connection to 192.168.24.8 closed. (undercloud) [stack@undercloud-0 ~]$ rpm -qa python3-glance-store openstack-tripleo-heat-templates puppet-tripleo puppet-glance puppet-tripleo-11.5.1-2.20210323024955.4d3d23e.el8ost.1.noarch puppet-glance-15.4.1-2.20210324004946.4b4b990.el8ost.1.noarch openstack-tripleo-heat-templates-11.4.1-2.20210323012110.c3396e2.el8ost.1.noarch (undercloud) [stack@undercloud-0 ~]$ rhos-release -L Installed repositories (rhel-8.4): 16.2 ceph-4 ceph-osd-4 rhel-8.4 (undercloud) [stack@undercloud-0 ~]$ cat /var/lib/rhos-release/latest-installed 16.2 -p RHOS-16.2-RHEL-8-20210409.n.0 (undercloud) [stack@undercloud-0 ~]$
1st test, rbd backend: ### verify instance is on rbd backend: (undercloud) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+--------------+--------+------------------------+----------------+------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+--------------+--------+------------------------+----------------+------------+ | 1ad66836-99b1-4e5d-ab78-e521a39780e9 | controller-2 | ACTIVE | ctlplane=192.168.24.44 | overcloud-full | controller | | 6749fcfb-ceff-4e3c-a618-b0d019dbb425 | controller-1 | ACTIVE | ctlplane=192.168.24.35 | overcloud-full | controller | | 92b0213c-ed0f-4afd-9b80-d72a52741920 | controller-0 | ACTIVE | ctlplane=192.168.24.8 | overcloud-full | controller | | 195a1d33-d0aa-477a-8b7c-c4c5c83b9cd3 | ceph-1 | ACTIVE | ctlplane=192.168.24.24 | overcloud-full | ceph | | 7470264a-7530-46ce-a5c4-9fd8102a360e | compute-1 | ACTIVE | ctlplane=192.168.24.53 | overcloud-full | compute | | 020f924d-4a11-4ab7-a119-b84dd0beac1b | ceph-0 | ACTIVE | ctlplane=192.168.24.32 | overcloud-full | ceph | | 3cd4d8fe-fcfe-4278-af6b-a50e58618796 | compute-0 | ACTIVE | ctlplane=192.168.24.22 | overcloud-full | compute | | 77b006d0-7565-4c83-abfe-5aa4a19028cc | ceph-2 | ACTIVE | ctlplane=192.168.24.30 | overcloud-full | ceph | +--------------------------------------+--------------+--------+------------------------+----------------+------------+ (undercloud) [stack@undercloud-0 ~]$ ssh -t heat-admin.24.8 "sudo podman exec -it -u root glance_api sh -c 'grep "enabled_backends" /etc/glance/glance-api.conf'" Warning: Permanently added '192.168.24.8' (ECDSA) to the list of known hosts. #enabled_backends = <None> enabled_backends=default_backend:rbd # by the ``enabled_backends`` configuration option in the DEFAULT # * enabled_backends # ``enabled_backends`` which helps to configure multiple backend stores WARN[0000] Error resizing exec session d934671a855074d842d3e61f9758d4db97cbacc7ccbd1af268db184d7619651e: could not open ctl file for terminal resize for container c98fe6f902bd7568075bf3f169c78bd43da74a3df2d2397948722e9435ae2454: open /var/lib/containers/storage/overlay-containers/c98fe6f902bd7568075bf3f169c78bd43da74a3df2d2397948722e9435ae2454/userdata/d934671a855074d842d3e61f9758d4db97cbacc7ccbd1af268db184d7619651e/ctl: no such device or address Connection to 192.168.24.8 closed. (undercloud) [stack@undercloud-0 ~]$ ### deploy config: (undercloud) [stack@undercloud-0 ~]$ cat glance_sparse_upload.yaml parameter_defaults: GlanceSparseUploadEnabled: true (undercloud) [stack@undercloud-0 ~]$ grep glance_sparse_upload.yaml overcloud_deploy.sh -e /home/stack/glance_sparse_upload.yaml \ (undercloud) [stack@undercloud-0 ~]$
=== check thin provisioning set to true: (undercloud) [stack@undercloud-0 ~]$ for i in 7 6 22; do echo "=== $i"; ssh -t heat-admin.24.$i "sudo podman exec -it -u root glance_api sh -c 'grep ^rbd_thin_provisioning /etc/glance/glance-api.conf'"; done === 7 Warning: Permanently added '192.168.24.7' (ECDSA) to the list of known hosts. WARN[0001] Error resizing exec session ea83283a48198ded805394ddd87c75c43ff78ebb12a278f9dc9c6ef5a78e22db: cannot resize container b274425b78bdb4ab921ec6851d278919f7580c6c456078552f6681027983e500 exec session ea83283a48198ded805394ddd87c75c43ff78ebb12a278f9dc9c6ef5a78e22db as it is not running: exec session state improper rbd_thin_provisioning=True Connection to 192.168.24.7 closed. === 6 Warning: Permanently added '192.168.24.6' (ECDSA) to the list of known hosts. rbd_thin_provisioning=True Connection to 192.168.24.6 closed. === 22 Warning: Permanently added '192.168.24.22' (ECDSA) to the list of known hosts. rbd_thin_provisioning=True WARN[0000] Error resizing exec session c16aef0934824fe179912d4af1ac0681b007111372c2f73d3803c0c4b6a35367: could not open ctl file for terminal resize for container 28412e3e176e10032519a372118e944a8269eb246f66ff1ad21e70d9afb24363: open /var/lib/containers/storage/overlay-containers/28412e3e176e10032519a372118e944a8269eb246f66ff1ad21e70d9afb24363/userdata/c16aef0934824fe179912d4af1ac0681b007111372c2f73d3803c0c4b6a35367/ctl: no such device or address Connection to 192.168.24.22 closed. (undercloud) [stack@undercloud-0 ~]$ === scenario with thin provisioning set to true: (overcloud) [stack@undercloud-0 ~]$ glance image-create-via-import --name Test-sparse1 --disk-format qcow2 --container-format bare --import-method web-download --uri https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1905.qcow2 (overcloud) [stack@undercloud-0 ~]$ glance image-list +--------------------------------------+----------------------------------+ | ID | Name | +--------------------------------------+----------------------------------+ | 831a8bee-151c-488f-bb5a-38db77147af9 | cirros-0.4.0-x86_64-disk.img | | cb9e7279-5650-4692-89f7-e56c862d4f45 | cirros-0.4.0-x86_64-disk.img_alt | | d22206d1-5097-492d-8478-fa2469b75e01 | Test-sparse1 | +--------------------------------------+----------------------------------+ === display raw image size (overcloud) [stack@undercloud-0 ~]$ qemu-img info CentOS-7-x86_64-GenericCloud-1905.qcow2 image: CentOS-7-x86_64-GenericCloud-1905.qcow2 file format: qcow2 virtual size: 8 GiB (8589934592 bytes) disk size: 898 MiB cluster_size: 65536 Format specific information: compat: 0.10 compression type: zlib refcount bits: 16 (overcloud) [stack@undercloud-0 ~]$ glance image-show d22206d1-5097-492d-8478-fa2469b75e01 | grep -i stores | os_glance_importing_to_stores | | | stores | default_backend | (overcloud) [stack@undercloud-0 ~]$ === display thin provisioning image size: [heat-admin@controller-0 ~]$ sudo rbd -p images diff d22206d1-5097-492d-8478-fa2469b75e01 | awk '{ SUM += $2 } END { print SUM/1024/1024/1024 " GB" }' 1.0625 GB [heat-admin@controller-0 ~]$
=== check thin provisioning set to false: (undercloud) [stack@undercloud-0 ~]$ for i in 7 6 22; do echo "=== $i"; ssh -t heat-admin.24.$i "sudo podman exec -it -u root glance_api sh -c 'grep ^rbd_thin_provisioning /etc/glance/glance-api.conf'"; done === 7 Warning: Permanently added '192.168.24.7' (ECDSA) to the list of known hosts. rbd_thin_provisioning=False WARN[0000] Error resizing exec session 5f5d9232b076a68ce171ba03fb87625fd70cd125a9278fc7a1c0105913f07970: could not open ctl file for terminal resize for container b274425b78bdb4ab921ec6851d278919f7580c6c456078552f6681027983e500: open /var/lib/containers/storage/overlay-containers/b274425b78bdb4ab921ec6851d278919f7580c6c456078552f6681027983e500/userdata/5f5d9232b076a68ce171ba03fb87625fd70cd125a9278fc7a1c0105913f07970/ctl: no such device or address Connection to 192.168.24.7 closed. === 6 Warning: Permanently added '192.168.24.6' (ECDSA) to the list of known hosts. rbd_thin_provisioning=False WARN[0000] Error resizing exec session 6662df0cf4ebea5a6d45bc2578240c8404c6a940d0bbc5c4b82a110e377669cc: could not open ctl file for terminal resize for container 7adb71be905249ef491f061c4bbae83c062bb70046aa3042716993c3d8f99566: open /var/lib/containers/storage/overlay-containers/7adb71be905249ef491f061c4bbae83c062bb70046aa3042716993c3d8f99566/userdata/6662df0cf4ebea5a6d45bc2578240c8404c6a940d0bbc5c4b82a110e377669cc/ctl: no such device or address Connection to 192.168.24.6 closed. === 22 Warning: Permanently added '192.168.24.22' (ECDSA) to the list of known hosts. rbd_thin_provisioning=False WARN[0000] Error resizing exec session 50f260d2d4e9a0577383eaf16ad0472b220f6bf02e0e38083c0cf62a32c53837: could not open ctl file for terminal resize for container 28412e3e176e10032519a372118e944a8269eb246f66ff1ad21e70d9afb24363: open /var/lib/containers/storage/overlay-containers/28412e3e176e10032519a372118e944a8269eb246f66ff1ad21e70d9afb24363/userdata/50f260d2d4e9a0577383eaf16ad0472b220f6bf02e0e38083c0cf62a32c53837/ctl: no such device or address Connection to 192.168.24.22 closed. (undercloud) [stack@undercloud-0 ~]$ === scenario with thin provisioning set to false: (overcloud) [stack@undercloud-0 ~]$ glance image-create-via-import --name Test-sparse-false-1 --disk-format qcow2 --container-format bare --import-method web-download --uri https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1905.qcow2 ... === display raw image size: (overcloud) [stack@undercloud-0 ~]$ qemu-img info CentOS-7-x86_64-GenericCloud-1905.qcow2 image: CentOS-7-x86_64-GenericCloud-1905.qcow2 file format: qcow2 virtual size: 8 GiB (8589934592 bytes) disk size: 898 MiB cluster_size: 65536 Format specific information: compat: 0.10 compression type: zlib refcount bits: 16 (overcloud) [stack@undercloud-0 ~]$ === display thin provisioning set to FALSE image size: [heat-admin@controller-0 ~]$ . ./overcloudrc (overcloud) [heat-admin@controller-0 ~]$ glance image-list +--------------------------------------+----------------------------------+ | ID | Name | +--------------------------------------+----------------------------------+ | 831a8bee-151c-488f-bb5a-38db77147af9 | cirros-0.4.0-x86_64-disk.img | | cb9e7279-5650-4692-89f7-e56c862d4f45 | cirros-0.4.0-x86_64-disk.img_alt | | 60b955e3-91b6-4e21-87d4-23175da590fb | Test-sparse-false-1 | +--------------------------------------+----------------------------------+ (overcloud) [heat-admin@controller-0 ~]$ sudo rbd -p images diff 60b955e3-91b6-4e21-87d4-23175da590fb | awk '{ SUM += $2 } END { print SUM/1024/1024/1024 " GB" }' 8 GB (overcloud) [heat-admin@controller-0 ~]$
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:3483
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days