Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1768441

Summary: [OSP13] Tempest run leaves orphaned snapshots
Product: Red Hat OpenStack Reporter: Lukasz Pelczyk <lpelczyk>
Component: openstack-tempestAssignee: Lukas Piwowarski <lpiwowar>
Status: CLOSED ERRATA QA Contact: Martin Kopec <mkopec>
Severity: low Docs Contact:
Priority: low    
Version: 13.0 (Queens)CC: apevec, lhh, slinaber, udesale
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-tempest-18.0.0-14.el7ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-06-24 11:41:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lukasz Pelczyk 2019-11-04 13:05:49 UTC
Description of problem:


Version-Release number of selected component (if applicable):
openstack 3.14.3 

How reproducible:
Every full run of tempest overcloud verification leaves nova-objects

Steps to Reproduce:
1. Perform full tempest run
2. Cleanup is performed after the run
3. There are instances snapshots left after the cleanup

Actual results:
Following objects were left uncleaned after the tempest run, causing red popup errors in Horizon: "Unable to retrieve project".
Problematic objects:
tempest-ServersNegativeTestJSON-server-1006268429-shelved
tempest-TestShelveInstance-server-1930908003-shelved
Both are instance snapshots, type: RAW, visibility: Image from Other Project - Non-Public, with deleted parent project.

Expected results:
Cleanup leaves no orphaned objects

Additional info:
N/A

Comment 1 Martin Kopec 2019-11-11 07:56:12 UTC
(overcloud) [stack@undercloud-0 ~]$ rpm -qa | grep tempest
python2-sahara-tests-tempest-0.6.0-2.el7ost.noarch
python2-cinder-tests-tempest-0.0.1-1.0.aafbdf8git.el7ost.noarch
python2-tempest-18.0.0-10.el7ost.noarch
python2-manila-tests-tempest-0.3.0-1.el7ost.noarch
openstack-tempest-18.0.0-10.el7ost.noarch
python2-sahara-tempest-0.6.0-2.el7ost.noarch
python2-neutron-tests-tempest-0.5.0-0.20190814141609.31993d5.el7ost.noarch
python2-tempestconf-2.3.0-1.el7ost.noarch

$ tempest cleanup --init-saved-state
$ openstack server create --image cirros-0.4.0-x86_64-disk.img --flavor  m1.micro --wait test
...
(overcloud) [stack@undercloud-0 cloud]$ openstack server stop test
(overcloud) [stack@undercloud-0 cloud]$ openstack server list
+--------------------------------------+------+---------+-----------------+------------------------------+----------+
| ID                                   | Name | Status  | Networks        | Image                        | Flavor   |
+--------------------------------------+------+---------+-----------------+------------------------------+----------+
| 330284b9-bd1f-44fb-92e1-52dcfe5b1467 | test | SHUTOFF | nova=10.0.0.210 | cirros-0.4.0-x86_64-disk.img | m1.micro |
+--------------------------------------+------+---------+-----------------+------------------------------+----------+
(overcloud) [stack@undercloud-0 cloud]$ openstack server image create --name test_snapshot test
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field            | Value                                                                                                                                                                                                                                                                          |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum         | None                                                                                                                                                                                                                                                                           |
| container_format | None                                                                                                                                                                                                                                                                           |
| created_at       | 2019-11-11T07:43:41Z                                                                                                                                                                                                                                                           |
| disk_format      | None                                                                                                                                                                                                                                                                           |
| file             | /v2/images/aaab0c9a-ec88-45c6-9d7e-d8cabddd0934/file                                                                                                                                                                                                                           |
| id               | aaab0c9a-ec88-45c6-9d7e-d8cabddd0934                                                                                                                                                                                                                                           |
| min_disk         | 1                                                                                                                                                                                                                                                                              |
| min_ram          | 0                                                                                                                                                                                                                                                                              |
| name             | test_snapshot                                                                                                                                                                                                                                                                  |
| owner            | 374de42a269d4004b572089bb54a5b46                                                                                                                                                                                                                                               |
| properties       | base_image_ref='f4e614ec-009b-4100-93ae-de0bb923b555', boot_roles='admin', hw_rng_model='virtio', image_type='snapshot', instance_uuid='330284b9-bd1f-44fb-92e1-52dcfe5b1467', owner_project_name='admin', owner_user_name='admin', user_id='1b2bae1244374468bf5c2ec3c7bdb382' |
| protected        | False                                                                                                                                                                                                                                                                          |
| schema           | /v2/schemas/image                                                                                                                                                                                                                                                              |
| size             | None                                                                                                                                                                                                                                                                           |
| status           | queued                                                                                                                                                                                                                                                                         |
| tags             |                                                                                                                                                                                                                                                                                |
| updated_at       | 2019-11-11T07:43:41Z                                                                                                                                                                                                                                                           |
| virtual_size     | None                                                                                                                                                                                                                                                                           |
| visibility       | private                                                                                                                                                                                                                                                                        |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 cloud]$ openstack image list
+--------------------------------------+----------------------------------+--------+
| ID                                   | Name                             | Status |
+--------------------------------------+----------------------------------+--------+
| f4e614ec-009b-4100-93ae-de0bb923b555 | cirros-0.4.0-x86_64-disk.img     | active |
| 23820532-31e7-4a30-a016-edec0738754d | cirros-0.4.0-x86_64-disk.img_alt | active |
| aaab0c9a-ec88-45c6-9d7e-d8cabddd0934 | test_snapshot                    | active |
+--------------------------------------+----------------------------------+--------+
(overcloud) [stack@undercloud-0 cloud]$

(overcloud) [stack@undercloud-0 cloud]$ tempest cleanup --dry-run
(overcloud) [stack@undercloud-0 cloud]$ cat dry_run.json | grep test_snapshot

(overcloud) [stack@undercloud-0 cloud]$ cp saved_state.json saved_state.json.first
(overcloud) [stack@undercloud-0 cloud]$ tempest cleanup --init-saved-state
Initializing saved state.
(overcloud) [stack@undercloud-0 cloud]$ cat saved_state.json | grep test_snapshot

Comment 10 Martin Kopec 2020-05-27 09:06:08 UTC
Installed 'Fixed in version' tempest package:

(overcloud) [stack@undercloud-0 tempest]$ openstack image list
+--------------------------------------+----------------------------------+--------+
| ID                                   | Name                             | Status |
+--------------------------------------+----------------------------------+--------+
| 5d31b3ea-3ebe-4ac7-83b6-bc2ebc414871 | cirros-0.4.0-x86_64-disk.img     | active |
| 95a48026-e24d-437f-8499-69d010341b3a | cirros-0.4.0-x86_64-disk.img_alt | active |
+--------------------------------------+----------------------------------+--------+
(overcloud) [stack@undercloud-0 tempest]$ openstack server create --image cirros-0.4.0-x86_64-disk.img --flavor  m1.micro --wait test

...

(overcloud) [stack@undercloud-0 tempest]$ openstack server stop test
(overcloud) [stack@undercloud-0 tempest]$ openstack server list
+--------------------------------------+------+---------+--------------------------------------------+------------------------------+----------+
| ID                                   | Name | Status  | Networks                                   | Image                        | Flavor   |
+--------------------------------------+------+---------+--------------------------------------------+------------------------------+----------+
| c7da0898-9113-4157-9c98-9eabe15ce1ea | test | SHUTOFF | public=2620:52:0:13b8::1000:9f, 10.0.0.220 | cirros-0.4.0-x86_64-disk.img | m1.micro |
+--------------------------------------+------+---------+--------------------------------------------+------------------------------+----------+
(overcloud) [stack@undercloud-0 tempest]$ server image create --name test_snapshot test
-bash: server: command not found
(overcloud) [stack@undercloud-0 tempest]$ openstack server image create --name test_snapshot test

...

(overcloud) [stack@undercloud-0 tempest]$ openstack image list
+--------------------------------------+----------------------------------+--------+
| ID                                   | Name                             | Status |
+--------------------------------------+----------------------------------+--------+
| 5d31b3ea-3ebe-4ac7-83b6-bc2ebc414871 | cirros-0.4.0-x86_64-disk.img     | active |
| 95a48026-e24d-437f-8499-69d010341b3a | cirros-0.4.0-x86_64-disk.img_alt | active |
| ed674b21-19e8-4c5e-814b-dba982267a1c | test_snapshot                    | active |
+--------------------------------------+----------------------------------+--------+
(overcloud) [stack@undercloud-0 tempest]$ tempest cleanup --dry-run
Begin cleanup
Process 0 projects
(overcloud) [stack@undercloud-0 tempest]$ cat dry_run.json | grep test_snapshot
      "name": "test_snapshot",
(overcloud) [stack@undercloud-0 tempest]$ cp saved_state.json saved_state.json.first
(overcloud) [stack@undercloud-0 tempest]$  tempest cleanup --init-saved-state
Initializing saved state.
(overcloud) [stack@undercloud-0 tempest]$ cat saved_state.json | grep test_snapshot
    "ed674b21-19e8-4c5e-814b-dba982267a1c": "test_snapshot"
(overcloud) [stack@undercloud-0 tempest]$ tempest cleanup 
Begin cleanup
Process 0 projects
(overcloud) [stack@undercloud-0 tempest]$ openstack image list
+--------------------------------------+----------------------------------+--------+
| ID                                   | Name                             | Status |
+--------------------------------------+----------------------------------+--------+
| 5d31b3ea-3ebe-4ac7-83b6-bc2ebc414871 | cirros-0.4.0-x86_64-disk.img     | active |
| 95a48026-e24d-437f-8499-69d010341b3a | cirros-0.4.0-x86_64-disk.img_alt | active |
| ed674b21-19e8-4c5e-814b-dba982267a1c | test_snapshot                    | active |
+--------------------------------------+----------------------------------+--------+
(overcloud) [stack@undercloud-0 tempest]$ cp saved_state.json.first saved_state.json
(overcloud) [stack@undercloud-0 tempest]$ tempest cleanup 
Begin cleanup
Process 0 projects
(overcloud) [stack@undercloud-0 tempest]$ openstack image list
+--------------------------------------+----------------------------------+--------+
| ID                                   | Name                             | Status |
+--------------------------------------+----------------------------------+--------+
| 5d31b3ea-3ebe-4ac7-83b6-bc2ebc414871 | cirros-0.4.0-x86_64-disk.img     | active |
| 95a48026-e24d-437f-8499-69d010341b3a | cirros-0.4.0-x86_64-disk.img_alt | active |
+--------------------------------------+----------------------------------+--------+

Comment 12 errata-xmlrpc 2020-06-24 11:41:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2719