Bug 1673103

Description of problem:

Removed networks after the tempest cleanup

Version-Release number of selected component (if applicable):

We have tempest 19.0.0.

rpm -qa | grep tempest
openstack-tempest-19.0.0-2.el7.noarch
python2-tempest-19.0.0-2.el7.noarch
python2-tempestconf-2.0.0-1.el7.noarch
puppet-tempest-13.3.1-1.el7.noarch

How reproducible:

I had two following networks available after the deployment.
$ openstack network list
+--------------------------------------+---------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+---------+--------------------------------------+
| 47bd1566-e13e-42b5-ad4b-fea122f425e9 | private | 98af2db8-2624-4471-8093-e983832e5dd9 |
| b121d356-2296-4100-9880-1a4410b4fd40 | public | 860fe620-6227-4349-9d8a-5947aa0f303b |
+--------------------------------------+---------+--------------------------------------+

After init-saved-state we can see, that there are no networks mentioned in the saved_state.json file - that's quite suspicious.
$ tempest cleanup --init-saved-state
$ cat saved_state.json
{
  "domains": {
    "default": "Default"
  },
  "flavors": {
    "1": "m1.tiny",
    "2": "m1.small",
    "3": "m1.medium",
    "4": "m1.large",
    "5": "m1.xlarge",
    "a83007fa-4b1b-4957-a731-1d82a6a264e0": "m1.micro",
    "dbc75455-e92d-4a3d-ace7-064ca8fc6324": "m1.nano"
  },
  "images": {},
  "projects": {
    "1efbbf4b524d44f694241e65dac11472": "alt_demo",
    "588d28c2d68e4857b72e7bd549141d55": "demo",
    "bc0183ea801d46ab869f2b40c15b2269": "admin",
    "c90f86114f444c09baa8a126fdf27f8b": "services"
  },
  "roles": {},
  "users": {
    "202c643912194efab607ab7b8e1f46da": "admin",
    "318fcff041f14bab9041759d54bdeecc": "cinder",
    "3b69696b13604b44a47475d383d65704": "nova",
    "534c5b16677044c1be2fe07d452d32dd": "aodh",
    "5ac1446caa9e4b6aa34d41dc7eb2f5d4": "neutron",
    "72cc57a76ea4477e8a33e88ca684d48a": "alt_demo",
    "75b2eb04cebf458ea8e489d16b2bb9b5": "demo",
    "8afab28d27ff466eaddb03bb423bd0a5": "swift",
    "aca4fcee47054713a1e91ee1ba277912": "glance",
    "b19acee3e3aa4c578848801f33c885b7": "gnocchi",
    "e07fcb5e10b245649d245a66d7451a96": "placement",
    "ebcd4caf6e274fb1a1eb395188233fb8": "ceilometer"
  }
}

Then I tried cleanup with --dry-run option and the generated file contained nothing to clean - that was good, but it was weird that the networks were not there because they weren't present in the saved_state.json, so does it mean that the `tempest cleanup` is not able to remove networks?
$ tempest cleanup --dry-run
$ cat dry_run.json
{
  "_projects_to_clean": {},
  "domains": [],
  "flavors": [],
  "images": [],
  "projects": [],
  "roles": [],
  "users": []
}

Then I did a little experiment and ran smoke tests:
$ tempest run --smoke
After a few seconds (let's say 30 sec) I killed the process, in order to make tempest left some networks and other resources behind.

$ openstack network list
+--------------------------------------+---------------------------------------------------------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+---------------------------------------------------------+--------------------------------------+
| 47bd1566-e13e-42b5-ad4b-fea122f425e9 | private | 98af2db8-2624-4471-8093-e983832e5dd9 |
| 69b82de6-2b24-4f15-b31c-64b304e4deb1 | tempest-ServersTestManualDisk-1308724128-network | e82f8798-b764-4556-9dcc-e0d08b20c02b |
| 8fe59e7d-89cb-4aed-82c4-4be235d38fdd | tempest-AttachInterfacesUnderV243Test-283077215-network | 28735895-9539-4475-b73b-70bd35ebe9c3 |
| b121d356-2296-4100-9880-1a4410b4fd40 | public | 860fe620-6227-4349-9d8a-5947aa0f303b |
| d3e43571-f9c6-48e0-99e6-f43f34c95581 | tempest-ServersTestBootFromVolume-577301809-network | 4e5db97f-b7fb-4fac-8de9-ae771f2d685a |
| edbe9f22-0dfb-44d3-ad43-497bbc8b0883 | tempest-ServerActionsTestJSON-293603464-network | 4cbf930e-dd9d-46f8-b7eb-5f15c4d09671 |
+--------------------------------------+---------------------------------------------------------+--------------------------------------+

I tried again dry cleanup:
$ tempest cleanup --dry-run

and I could see that even my private network was present in the dry_run.json file - but why now? Before it wasn't marked as a leftover resource (even it should have, because it was not in the saved_state.json), but now yes.

And after I did the real cleanup, the network was gone.
$ openstack network list
+--------------------------------------+--------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+--------+--------------------------------------+
| b121d356-2296-4100-9880-1a4410b4fd40 | public | 860fe620-6227-4349-9d8a-5947aa0f303b |
+--------------------------------------+--------+--------------------------------------+