Bug 1931979
| Summary: | Glance will leak staging data | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Abhishek Kekane <akekane> | |
| Component: | openstack-glance | Assignee: | Dan Smith <dasmith> | |
| Status: | CLOSED ERRATA | QA Contact: | msava | |
| Severity: | medium | Docs Contact: | RHOS Documentation Team <rhos-docs> | |
| Priority: | high | |||
| Version: | 17.0 (Wallaby) | CC: | athomas, cyril, eglynn, gcharot, jschluet, msava, tshefi, udesale, vimartin | |
| Target Milestone: | Alpha | Keywords: | Triaged | |
| Target Release: | 17.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-glance-22.1.1-0.20220221192913.b1d635c.el9ost | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1945634 (view as bug list) | Environment: | ||
| Last Closed: | 2022-09-21 12:14:01 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1945634 | |||
|
Description
Abhishek Kekane
2021-02-23 17:19:07 UTC
OK, once it's merged upstream, I'll look into backporting this to 16.2 and 16.1. @Maxim: can we see the full glance-api.conf and the logs? This is a multi-store deployment, right? Tests failed according to Abishek scenarios.
Tested environment: with 3 controllers
Scenario1:
1.
+--------------------------------------+---------------------------------------------+ [11/881]
(overcloud) [stack@undercloud-0 ~]$ glance image-create-via-import --container-format bare --name debian_web_download --import-method web-download --disk-format raw --uri https
://cloud.debian.org/cdimage/cloud/OpenStack/current-10/debian-10-openstack-arm64.raw
+-----------------------+--------------------------------------+
| Property | Value |
+-----------------------+--------------------------------------+
| checksum | None |
| container_format | bare |
| created_at | 2022-07-17T09:02:34Z |
| disk_format | raw |
| id | d42ba4db-2b48-4fd8-96a5-46fbb089bc5c |
| min_disk | 0 |
| min_ram | 0 |
| name | debian_web_download |
| os_glance_import_task | 190d3f6a-4aff-4c5a-85bd-80675bec30a9 |
| os_hash_algo | None |
| os_hash_value | None |
| os_hidden | False |
| owner | e09c9609aa9c439fa84f5e5748627d2d |
| protected | False |
| size | None |
status | queued |
| tags | [] |
| updated_at | 2022-07-17T09:02:34Z |
| virtual_size | Not available |
| visibility | shared |
+-----------------------+--------------------------------------+
2. sudo podman stop glance_api
3.(overcloud) [stack@undercloud-0 ~]$ glance image-show d42ba4db-2b48-4fd8-96a5-46fbb089bc5c [7/909]
+-------------------------------+--------------------------------------+
| Property | Value |
+-------------------------------+--------------------------------------+
| checksum | None |
| container_format | bare |
| created_at | 2022-07-17T09:02:34Z |
| disk_format | raw |
| id | d42ba4db-2b48-4fd8-96a5-46fbb089bc5c |
| min_disk | 0 |
| min_ram | 0 |
| name | debian_web_download |
| os_glance_failed_import | |
| os_glance_import_task | 190d3f6a-4aff-4c5a-85bd-80675bec30a9 |
| os_glance_importing_to_stores | default_backend |
| os_hash_algo | None |
| os_hash_value | None |
| os_hidden | False |
| owner | e09c9609aa9c439fa84f5e5748627d2d |
| protected | False |
| size | None |
| status | importing |
| tags | [] |
| updated_at | 2022-07-17T09:02:35Z |
| virtual_size | Not available |
| visibility | shared |
+-------------------------------+--------------------------------------+
2. [heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store
total 190400
drwxr-x---. 2 42415 42415 50 Jul 17 09:02 .
drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 ..
-rw-r-----. 1 42415 42415 194969600 Jul 17 09:02 d42ba4db-2b48-4fd8-96a5-46fbb089bc5c
3.(overcloud) [stack@undercloud-0 ~]$ glance image-delete d42ba4db-2b48-4fd8-96a5-46fbb089bc5c
4.sudo ls -la /var/lib/glance/staging_store
total 190400
drwxr-x---. 2 42415 42415 50 Jul 17 09:02 .
drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 ..
-rw-r-----. 1 42415 42415 194969600 Jul 17 09:02 d42ba4db-2b48-4fd8-96a5-46fbb089bc5c
5. Glance image-delete
6.Image still in staging
heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store
total 190400
drwxr-x---. 2 42415 42415 50 Jul 17 09:02 .
drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 ..
-rw-r-----. 1 42415 42415 194969600 Jul 17 09:02 d42ba4db-2b48-4fd8-96a5-46fbb089bc5c
But if we again stop,start the image deleted.
Scenario2:
1.
(overcloud) [stack@undercloud-0 ~]$ glance image-create-via-import --container-format bare --name debian_web_download --import-method web-download --disk-format raw --uri https
://cloud.debian.org/cdimage/cloud/OpenStack/current-10/debian-10-openstack-arm64.raw
+-----------------------+--------------------------------------+
| Property | Value |
+-----------------------+--------------------------------------+
| checksum | None |
| container_format | bare |
| created_at | 2022-07-17T09:23:24Z |
| disk_format | raw |
| id | 6f9a7917-a883-414f-98f6-7ad6379dc8c8 |
| min_disk | 0 |
| min_ram | 0 |
| name | debian_web_download |
| os_glance_import_task | df38eb3c-3a78-41a4-9e50-ea64d34a70c1 |
| os_hash_algo | None |
| os_hash_value | None |
| os_hidden | False |
| owner | e09c9609aa9c439fa84f5e5748627d2d |
| protected | False |
| size | None |
| status | queued |
| tags | [] |
| updated_at | 2022-07-17T09:23:24Z |
| virtual_size | Not available |
| visibility | shared |
+-----------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ glance image-show 6f9a7917-a883-414f-98f6-7ad6379dc8c8 [7/999]
+-------------------------------+--------------------------------------+
| Property | Value |
+-------------------------------+--------------------------------------+
| checksum | None |
| container_format | bare |
| created_at | 2022-07-17T09:23:24Z |
| disk_format | raw |
| id | 6f9a7917-a883-414f-98f6-7ad6379dc8c8 |
| min_disk | 0 |
| min_ram | 0 |
| name | debian_web_download |
| os_glance_failed_import | |
| os_glance_import_task | df38eb3c-3a78-41a4-9e50-ea64d34a70c1 |
| os_glance_importing_to_stores | default_backend |
| os_hash_algo | None |
| os_hash_value | None |
| os_hidden | False |
| owner | e09c9609aa9c439fa84f5e5748627d2d |
| protected | False |
| size | None |
2.(overcloud) [stack@undercloud-0 ~]$ ctlplane-1 sudo podman stop glance_api
3.[heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store
total 46080
drwxr-x---. 2 42415 42415 50 Jul 17 09:23 .
drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 ..
-rw-r-----. 1 42415 42415 47185920 Jul 17 09:23 6f9a7917-a883-414f-98f6-7ad6379dc8c8
4.[heat-admin@controller-1 ~]$ sudo podman start glance_api
5.
[heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store
total 46080
drwxr-x---. 2 42415 42415 50 Jul 17 09:23 .
drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 ..
-rw-r-----. 1 42415 42415 47185920 Jul 17 09:23 6f9a7917-a883-414f-98f6-7ad6379dc8c8
6.(overcloud) [stack@undercloud-0 ~]$ glance image-delete 6f9a7917-a883-414f-98f6-7ad6379dc8c8
7.image still in staging directory
heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store
total 46080
drwxr-x---. 2 42415 42415 50 Jul 17 09:23 .
drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 ..
-rw-r-----. 1 42415 42415 47185920 Jul 17 09:23 6f9a7917-a883-414f-98f6-7ad6379dc8c8
But if we again start,stop glance_api the image deleted.
(In reply to msava from comment #11) > Tests failed according to Abishek scenarios. > Tested environment: with 3 controllers > Scenario1: > 1. > +--------------------------------------+------------------------------------- > --------+ > [11/881] > (overcloud) [stack@undercloud-0 ~]$ glance image-create-via-import > --container-format bare --name debian_web_download --import-method > web-download --disk-format raw --uri https > ://cloud.debian.org/cdimage/cloud/OpenStack/current-10/debian-10-openstack- > arm64.raw > > +-----------------------+--------------------------------------+ > > | Property | Value | > > +-----------------------+--------------------------------------+ > > | checksum | None | > > | container_format | bare | > > | created_at | 2022-07-17T09:02:34Z | > > | disk_format | raw | > > | id | d42ba4db-2b48-4fd8-96a5-46fbb089bc5c | > > | min_disk | 0 | > > | min_ram | 0 | > | name | debian_web_download | > | os_glance_import_task | 190d3f6a-4aff-4c5a-85bd-80675bec30a9 | > | os_hash_algo | None | > | os_hash_value | None | > | os_hidden | False | > | owner | e09c9609aa9c439fa84f5e5748627d2d | > | protected | False | > | size | None | > status | queued | > | tags | [] | > | updated_at | 2022-07-17T09:02:34Z | > | virtual_size | Not available | > | visibility | shared | > +-----------------------+--------------------------------------+ > > 2. sudo podman stop glance_api > 3.(overcloud) [stack@undercloud-0 ~]$ glance image-show > d42ba4db-2b48-4fd8-96a5-46fbb089bc5c > [7/909] > +-------------------------------+--------------------------------------+ > | Property | Value | > +-------------------------------+--------------------------------------+ > | checksum | None | > | container_format | bare | > | created_at | 2022-07-17T09:02:34Z | > | disk_format | raw | > | id | d42ba4db-2b48-4fd8-96a5-46fbb089bc5c | > | min_disk | 0 | > | min_ram | 0 | > | name | debian_web_download | > | os_glance_failed_import | | > | os_glance_import_task | 190d3f6a-4aff-4c5a-85bd-80675bec30a9 | > | os_glance_importing_to_stores | default_backend | > | os_hash_algo | None | > | os_hash_value | None | > | os_hidden | False | > | owner | e09c9609aa9c439fa84f5e5748627d2d | > | protected | False | > | size | None | > | status | importing | > | tags | [] | > | updated_at | 2022-07-17T09:02:35Z | > | virtual_size | Not available | > | visibility | shared | > +-------------------------------+--------------------------------------+ > > > > 2. [heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store > total 190400 > drwxr-x---. 2 42415 42415 50 Jul 17 09:02 . > drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. > -rw-r-----. 1 42415 42415 194969600 Jul 17 09:02 > d42ba4db-2b48-4fd8-96a5-46fbb089bc5c > > 3.(overcloud) [stack@undercloud-0 ~]$ glance image-delete > d42ba4db-2b48-4fd8-96a5-46fbb089bc5c > > 4.sudo ls -la /var/lib/glance/staging_store > total 190400 > drwxr-x---. 2 42415 42415 50 Jul 17 09:02 . > > drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. > > -rw-r-----. 1 42415 42415 194969600 Jul 17 09:02 > d42ba4db-2b48-4fd8-96a5-46fbb089bc5c > > 5. Glance image-delete > > 6.Image still in staging > heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store > total 190400 > drwxr-x---. 2 42415 42415 50 Jul 17 09:02 . > drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. > -rw-r-----. 1 42415 42415 194969600 Jul 17 09:02 > d42ba4db-2b48-4fd8-96a5-46fbb089bc5c > > > But if we again stop,start the image deleted. > > > Scenario2: > 1. > (overcloud) [stack@undercloud-0 ~]$ glance image-create-via-import > --container-format bare --name debian_web_download --import-method > web-download --disk-format raw --uri https > ://cloud.debian.org/cdimage/cloud/OpenStack/current-10/debian-10-openstack- > arm64.raw > > +-----------------------+--------------------------------------+ > > | Property | Value | > > +-----------------------+--------------------------------------+ > > | checksum | None | > > | container_format | bare | > > | created_at | 2022-07-17T09:23:24Z | > > | disk_format | raw | > > | id | 6f9a7917-a883-414f-98f6-7ad6379dc8c8 | > > | min_disk | 0 | > > | min_ram | 0 | > | name | debian_web_download | > | os_glance_import_task | df38eb3c-3a78-41a4-9e50-ea64d34a70c1 | > | os_hash_algo | None | > | os_hash_value | None | > | os_hidden | False | > | owner | e09c9609aa9c439fa84f5e5748627d2d | > | protected | False | > | size | None | > | status | queued | > | tags | [] | > | updated_at | 2022-07-17T09:23:24Z | > | virtual_size | Not available | > | visibility | shared | > +-----------------------+--------------------------------------+ > > (overcloud) [stack@undercloud-0 ~]$ glance image-show > 6f9a7917-a883-414f-98f6-7ad6379dc8c8 > [7/999] > +-------------------------------+--------------------------------------+ > | Property | Value | > +-------------------------------+--------------------------------------+ > | checksum | None | > | container_format | bare | > | created_at | 2022-07-17T09:23:24Z | > | disk_format | raw | > | id | 6f9a7917-a883-414f-98f6-7ad6379dc8c8 | > | min_disk | 0 | > | min_ram | 0 | > | name | debian_web_download | > | os_glance_failed_import | | > | os_glance_import_task | df38eb3c-3a78-41a4-9e50-ea64d34a70c1 | > | os_glance_importing_to_stores | default_backend | > | os_hash_algo | None | > | os_hash_value | None | > | os_hidden | False | > | owner | e09c9609aa9c439fa84f5e5748627d2d | > | protected | False | > | size | None | > > 2.(overcloud) [stack@undercloud-0 ~]$ ctlplane-1 sudo podman stop glance_api > 3.[heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store > > total 46080 > drwxr-x---. 2 42415 42415 50 Jul 17 09:23 . > > drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. > > -rw-r-----. 1 42415 42415 47185920 Jul 17 09:23 > 6f9a7917-a883-414f-98f6-7ad6379dc8c8 > > 4.[heat-admin@controller-1 ~]$ sudo podman start glance_api > 5. > [heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store > > total 46080 > drwxr-x---. 2 42415 42415 50 Jul 17 09:23 . > > drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. > > -rw-r-----. 1 42415 42415 47185920 Jul 17 09:23 > 6f9a7917-a883-414f-98f6-7ad6379dc8c8 > > 6.(overcloud) [stack@undercloud-0 ~]$ glance image-delete > 6f9a7917-a883-414f-98f6-7ad6379dc8c8 > > 7.image still in staging directory > heat-admin@controller-1 ~]$ sudo ls -la /var/lib/glance/staging_store > > total 46080 > drwxr-x---. 2 42415 42415 50 Jul 17 09:23 . > > drwxr-xr-x. 7 42415 42415 116 Jul 17 06:25 .. > > -rw-r-----. 1 42415 42415 47185920 Jul 17 09:23 > 6f9a7917-a883-414f-98f6-7ad6379dc8c8 > > But if we again start,stop glance_api the image deleted. Hi Maxima, Staging area is cleared means code is working and verified. I have written above scenarios considering all in one controller node. In case of 3 controller nodes we does not guarantee that delete image call will go to same controller where staging data is present and that is why you need to restart the service on that controller node where staging data is present. So in case of multiple controller nodes for both the scenarios add one more step at the in the end to "restart g-api service on all controller nodes". Abhishek VERIFIED Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2022:6543 |