Description of problem: In various situations, glance will leak (potentially very large) temporary files in the staging store. One example is doing a web-download import, where glance initially downloads the image to its staging store. If the worker doing that activity crashes, loses power, etc, the user may delete the image and try again on another worker. When the crashed worker resumes, the staging data will remain but nothing will ever clean it up. Another example would be a misconfigured glance that uses local staging directories, but glance-direct is used, where the user stages data, and then deletes the image from another worker. Even in a situation where shared staging is properly configured, a failure to access the staging location during the delete call will result in the image being deleted, but the staging file not being purged. IMHO, glance workers should clean their staging directories at startup, purging any data that is attributable to a previous image having been deleted. Another option is to add a store location for each staged image, and make sure the scrubber can clean those things from the staging directory periodically (this requires also running the scrubber on each node, which may not be common practice currently). Note: We can also backport it to previous version i.e. OSP 16
OK, once it's merged upstream, I'll look into backporting this to 16.2 and 16.1.
@Maxim: can we see the full glance-api.conf and the logs? This is a multi-store deployment, right?
Tests failed according to Abishek scenarios. Tested environment: with 3 controllers Scenario1: 1. +--------------------------------------+---------------------------------------------+ [11/881] (overcloud) [stack@undercloud-0 ~]$ glance image-create-via-import --container-format bare --name debian_web_download --import-method web-download --disk-format raw --uri https ://cloud.debian.org/cdimage/cloud/OpenStack/current-10/debian-10-openstack-arm64.raw +-----------------------+--------------------------------------+ | Property | Value | +-----------------------+--------------------------------------+ | checksum | None | | container_format | bare | | created_at | 2022-07-17T09:02:34Z | | disk_format | raw | | id | d42ba4db-2b48-4fd8-96a5-46fbb089bc5c | | min_disk | 0 | | min_ram | 0 | | name | debian_web_download | | os_glance_import_task | 190d3f6a-4aff-4c5a-85bd-80675bec30a9 | | os_hash_algo | None | | os_hash_value | None | | os_hidden | False | | owner | e09c9609aa9c439fa84f5e5748627d2d | | protected | False | | size | None | status | queued | | tags | [] | | updated_at | 2022-07-17T09:02:34Z | | virtual_size | Not available | | visibility | shared | +-----------------------+--------------------------------------+ 2. sudo podman stop glance_api 3.(overcloud) [stack@undercloud-0 ~]$ glance image-show d42ba4db-2b48-4fd8-96a5-46fbb089bc5c [7/909] +-------------------------------+--------------------------------------+ | Property | Value | +-------------------------------+--------------------------------------+ | checksum | None | | container_format | bare | | created_at | 2022-07-17T09:02:34Z | | disk_format | raw | | id | d42ba4db-2b48-4fd8-96a5-46fbb089bc5c | | min_disk | 0 | | min_ram | 0 | | name | debian_web_download | | os_glance_failed_import | | | os_glance_import_task | 190d3f6a-4aff-4c5a-85bd-80675bec30a9 | | os_glance_importing_to_stores | default_backend | | os_hash_algo | None | | os_hash_value | None | | os_hidden | False | | owner | e09c9609aa9c439fa84f5e5748627d2d | | protected | False | | size | None | | status | importing | | tags | [] | | updated_at | 2022-07-17T09:02:35Z | | virtual_size | Not available | | visibility | shared | +-------------------------------+--------------------------------------+ 2. [heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store total 190400 drwxr-x---. 2 42415 42415 50 Jul 17 09:02 . drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. -rw-r-----. 1 42415 42415 194969600 Jul 17 09:02 d42ba4db-2b48-4fd8-96a5-46fbb089bc5c 3.(overcloud) [stack@undercloud-0 ~]$ glance image-delete d42ba4db-2b48-4fd8-96a5-46fbb089bc5c 4.sudo ls -la /var/lib/glance/staging_store total 190400 drwxr-x---. 2 42415 42415 50 Jul 17 09:02 . drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. -rw-r-----. 1 42415 42415 194969600 Jul 17 09:02 d42ba4db-2b48-4fd8-96a5-46fbb089bc5c 5. Glance image-delete 6.Image still in staging heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store total 190400 drwxr-x---. 2 42415 42415 50 Jul 17 09:02 . drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. -rw-r-----. 1 42415 42415 194969600 Jul 17 09:02 d42ba4db-2b48-4fd8-96a5-46fbb089bc5c But if we again stop,start the image deleted. Scenario2: 1. (overcloud) [stack@undercloud-0 ~]$ glance image-create-via-import --container-format bare --name debian_web_download --import-method web-download --disk-format raw --uri https ://cloud.debian.org/cdimage/cloud/OpenStack/current-10/debian-10-openstack-arm64.raw +-----------------------+--------------------------------------+ | Property | Value | +-----------------------+--------------------------------------+ | checksum | None | | container_format | bare | | created_at | 2022-07-17T09:23:24Z | | disk_format | raw | | id | 6f9a7917-a883-414f-98f6-7ad6379dc8c8 | | min_disk | 0 | | min_ram | 0 | | name | debian_web_download | | os_glance_import_task | df38eb3c-3a78-41a4-9e50-ea64d34a70c1 | | os_hash_algo | None | | os_hash_value | None | | os_hidden | False | | owner | e09c9609aa9c439fa84f5e5748627d2d | | protected | False | | size | None | | status | queued | | tags | [] | | updated_at | 2022-07-17T09:23:24Z | | virtual_size | Not available | | visibility | shared | +-----------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ glance image-show 6f9a7917-a883-414f-98f6-7ad6379dc8c8 [7/999] +-------------------------------+--------------------------------------+ | Property | Value | +-------------------------------+--------------------------------------+ | checksum | None | | container_format | bare | | created_at | 2022-07-17T09:23:24Z | | disk_format | raw | | id | 6f9a7917-a883-414f-98f6-7ad6379dc8c8 | | min_disk | 0 | | min_ram | 0 | | name | debian_web_download | | os_glance_failed_import | | | os_glance_import_task | df38eb3c-3a78-41a4-9e50-ea64d34a70c1 | | os_glance_importing_to_stores | default_backend | | os_hash_algo | None | | os_hash_value | None | | os_hidden | False | | owner | e09c9609aa9c439fa84f5e5748627d2d | | protected | False | | size | None | 2.(overcloud) [stack@undercloud-0 ~]$ ctlplane-1 sudo podman stop glance_api 3.[heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store total 46080 drwxr-x---. 2 42415 42415 50 Jul 17 09:23 . drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. -rw-r-----. 1 42415 42415 47185920 Jul 17 09:23 6f9a7917-a883-414f-98f6-7ad6379dc8c8 4.[heat-admin@controller-1 ~]$ sudo podman start glance_api 5. [heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store total 46080 drwxr-x---. 2 42415 42415 50 Jul 17 09:23 . drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. -rw-r-----. 1 42415 42415 47185920 Jul 17 09:23 6f9a7917-a883-414f-98f6-7ad6379dc8c8 6.(overcloud) [stack@undercloud-0 ~]$ glance image-delete 6f9a7917-a883-414f-98f6-7ad6379dc8c8 7.image still in staging directory heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store total 46080 drwxr-x---. 2 42415 42415 50 Jul 17 09:23 . drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. -rw-r-----. 1 42415 42415 47185920 Jul 17 09:23 6f9a7917-a883-414f-98f6-7ad6379dc8c8 But if we again start,stop glance_api the image deleted.
(In reply to msava from comment #11) > Tests failed according to Abishek scenarios. > Tested environment: with 3 controllers > Scenario1: > 1. > +--------------------------------------+------------------------------------- > --------+ > [11/881] > (overcloud) [stack@undercloud-0 ~]$ glance image-create-via-import > --container-format bare --name debian_web_download --import-method > web-download --disk-format raw --uri https > ://cloud.debian.org/cdimage/cloud/OpenStack/current-10/debian-10-openstack- > arm64.raw > > +-----------------------+--------------------------------------+ > > | Property | Value | > > +-----------------------+--------------------------------------+ > > | checksum | None | > > | container_format | bare | > > | created_at | 2022-07-17T09:02:34Z | > > | disk_format | raw | > > | id | d42ba4db-2b48-4fd8-96a5-46fbb089bc5c | > > | min_disk | 0 | > > | min_ram | 0 | > | name | debian_web_download | > | os_glance_import_task | 190d3f6a-4aff-4c5a-85bd-80675bec30a9 | > | os_hash_algo | None | > | os_hash_value | None | > | os_hidden | False | > | owner | e09c9609aa9c439fa84f5e5748627d2d | > | protected | False | > | size | None | > status | queued | > | tags | [] | > | updated_at | 2022-07-17T09:02:34Z | > | virtual_size | Not available | > | visibility | shared | > +-----------------------+--------------------------------------+ > > 2. sudo podman stop glance_api > 3.(overcloud) [stack@undercloud-0 ~]$ glance image-show > d42ba4db-2b48-4fd8-96a5-46fbb089bc5c > [7/909] > +-------------------------------+--------------------------------------+ > | Property | Value | > +-------------------------------+--------------------------------------+ > | checksum | None | > | container_format | bare | > | created_at | 2022-07-17T09:02:34Z | > | disk_format | raw | > | id | d42ba4db-2b48-4fd8-96a5-46fbb089bc5c | > | min_disk | 0 | > | min_ram | 0 | > | name | debian_web_download | > | os_glance_failed_import | | > | os_glance_import_task | 190d3f6a-4aff-4c5a-85bd-80675bec30a9 | > | os_glance_importing_to_stores | default_backend | > | os_hash_algo | None | > | os_hash_value | None | > | os_hidden | False | > | owner | e09c9609aa9c439fa84f5e5748627d2d | > | protected | False | > | size | None | > | status | importing | > | tags | [] | > | updated_at | 2022-07-17T09:02:35Z | > | virtual_size | Not available | > | visibility | shared | > +-------------------------------+--------------------------------------+ > > > > 2. [heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store > total 190400 > drwxr-x---. 2 42415 42415 50 Jul 17 09:02 . > drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. > -rw-r-----. 1 42415 42415 194969600 Jul 17 09:02 > d42ba4db-2b48-4fd8-96a5-46fbb089bc5c > > 3.(overcloud) [stack@undercloud-0 ~]$ glance image-delete > d42ba4db-2b48-4fd8-96a5-46fbb089bc5c > > 4.sudo ls -la /var/lib/glance/staging_store > total 190400 > drwxr-x---. 2 42415 42415 50 Jul 17 09:02 . > > drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. > > -rw-r-----. 1 42415 42415 194969600 Jul 17 09:02 > d42ba4db-2b48-4fd8-96a5-46fbb089bc5c > > 5. Glance image-delete > > 6.Image still in staging > heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store > total 190400 > drwxr-x---. 2 42415 42415 50 Jul 17 09:02 . > drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. > -rw-r-----. 1 42415 42415 194969600 Jul 17 09:02 > d42ba4db-2b48-4fd8-96a5-46fbb089bc5c > > > But if we again stop,start the image deleted. > > > Scenario2: > 1. > (overcloud) [stack@undercloud-0 ~]$ glance image-create-via-import > --container-format bare --name debian_web_download --import-method > web-download --disk-format raw --uri https > ://cloud.debian.org/cdimage/cloud/OpenStack/current-10/debian-10-openstack- > arm64.raw > > +-----------------------+--------------------------------------+ > > | Property | Value | > > +-----------------------+--------------------------------------+ > > | checksum | None | > > | container_format | bare | > > | created_at | 2022-07-17T09:23:24Z | > > | disk_format | raw | > > | id | 6f9a7917-a883-414f-98f6-7ad6379dc8c8 | > > | min_disk | 0 | > > | min_ram | 0 | > | name | debian_web_download | > | os_glance_import_task | df38eb3c-3a78-41a4-9e50-ea64d34a70c1 | > | os_hash_algo | None | > | os_hash_value | None | > | os_hidden | False | > | owner | e09c9609aa9c439fa84f5e5748627d2d | > | protected | False | > | size | None | > | status | queued | > | tags | [] | > | updated_at | 2022-07-17T09:23:24Z | > | virtual_size | Not available | > | visibility | shared | > +-----------------------+--------------------------------------+ > > (overcloud) [stack@undercloud-0 ~]$ glance image-show > 6f9a7917-a883-414f-98f6-7ad6379dc8c8 > [7/999] > +-------------------------------+--------------------------------------+ > | Property | Value | > +-------------------------------+--------------------------------------+ > | checksum | None | > | container_format | bare | > | created_at | 2022-07-17T09:23:24Z | > | disk_format | raw | > | id | 6f9a7917-a883-414f-98f6-7ad6379dc8c8 | > | min_disk | 0 | > | min_ram | 0 | > | name | debian_web_download | > | os_glance_failed_import | | > | os_glance_import_task | df38eb3c-3a78-41a4-9e50-ea64d34a70c1 | > | os_glance_importing_to_stores | default_backend | > | os_hash_algo | None | > | os_hash_value | None | > | os_hidden | False | > | owner | e09c9609aa9c439fa84f5e5748627d2d | > | protected | False | > | size | None | > > 2.(overcloud) [stack@undercloud-0 ~]$ ctlplane-1 sudo podman stop glance_api > 3.[heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store > > total 46080 > drwxr-x---. 2 42415 42415 50 Jul 17 09:23 . > > drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. > > -rw-r-----. 1 42415 42415 47185920 Jul 17 09:23 > 6f9a7917-a883-414f-98f6-7ad6379dc8c8 > > 4.[heat-admin@controller-1 ~]$ sudo podman start glance_api > 5. > [heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store > > total 46080 > drwxr-x---. 2 42415 42415 50 Jul 17 09:23 . > > drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. > > -rw-r-----. 1 42415 42415 47185920 Jul 17 09:23 > 6f9a7917-a883-414f-98f6-7ad6379dc8c8 > > 6.(overcloud) [stack@undercloud-0 ~]$ glance image-delete > 6f9a7917-a883-414f-98f6-7ad6379dc8c8 > > 7.image still in staging directory > heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store > > total 46080 > drwxr-x---. 2 42415 42415 50 Jul 17 09:23 . > > drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. > > -rw-r-----. 1 42415 42415 47185920 Jul 17 09:23 > 6f9a7917-a883-414f-98f6-7ad6379dc8c8 > > But if we again start,stop glance_api the image deleted. Hi Maxima, Staging area is cleared means code is working and verified. I have written above scenarios considering all in one controller node. In case of 3 controller nodes we does not guarantee that delete image call will go to same controller where staging data is present and that is why you need to restart the service on that controller node where staging data is present. So in case of multiple controller nodes for both the scenarios add one more step at the in the end to "restart g-api service on all controller nodes". Abhishek
VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2022:6543