$ tempest cleanup --init-saved-state 2019-10-11 13:20:49.606 4593 INFO tempest.lib.common.rest_client [req-279f9f8b-070a-49d5-9fbb-9a85af8c8129 ] Request (main): 404 GET https://192.XXX.XX.X:XXXXX/v2.0/floatingips 0.239s 2019-10-11 13:20:49.607 4593 DEBUG tempest.lib.common.rest_client [req-279f9f8b-070a-49d5-9fbb-9a85af8c8129 ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'} Body: None Response - Headers: {'status': '404', u'content-length': '103', 'content-location': 'https://192.XXX.XX.X:XXXXX/v2.0/floatingips', u'date': 'Fri, 11 Oct 2019 17:20:49 GMT', u'content-type': 'application/json', u'connection': 'close', u'x-openstack-request-id': 'req-279f9f8b-070a-49d5-9fbb-9a85af8c8129'} Body: {"NeutronError": {"message": "The resource could not be found.", "type": "HTTPNotFound", "detail": ""}} _log_request_full /usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py:434 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup [-] Failure during cleanup: NotFound: Object not found Details: {u'message': u'The resource could not be found.', u'type': u'HTTPNotFound', u'detail': u''} 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup Traceback (most recent call last): 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 110, in take_action 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup self.init(parsed_args) 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 146, in init 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup self._init_state() 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 332, in _init_state 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup svc.run() 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup_service.py", line 134, in run 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup self.save_state() 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup_service.py", line 452, in save_state 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup flips = self.list() 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup_service.py", line 428, in list 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup flips = client.list_floatingips(**self.tenant_filter) 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/floating_ips_client.py", line 65, in list_floatingips 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup return self.list_resources(uri, **filters) 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/base.py", line 34, in list_resources 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup resp, body = self.get(req_uri) 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 294, in get 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup return self.request('GET', url, extra_headers, headers) 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup self._error_checker(resp, resp_body) 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 774, in _error_checker 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup raise exceptions.NotFound(resp_body, resp=resp) 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup NotFound: Object not found 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup Details: {u'message': u'The resource could not be found.', u'type': u'HTTPNotFound', u'detail': u''} 2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup 2019-10-11 13:20:49.609 4593 ERROR tempest [-] Object not found Details: {u'message': u'The resource could not be found.', u'type': u'HTTPNotFound', u'detail': u''}: NotFound: Object not found 2019-10-11 13:20:49.609 4593 DEBUG tempest.cmd.main [-] tempest clean_up TempestCleanup clean_up /usr/lib/python2.7/site-packages/tempest/cmd/main.py:41 2019-10-11 13:20:49.610 4593 DEBUG tempest.cmd.main [-] tempest got an error: Object not found Details: {u'message': u'The resource could not be found.', u'type': u'HTTPNotFound', u'detail': u''} clean_up /usr/lib/python2.7/site-packages/tempest/cmd/main.py:43
I reproduced the issue with tempest.conf which contained the following misconfigurations: 1. admin_password doesn't belong to [identity] section anymore [1], it should be under [auth] section. 2. I noticed that admin_password and uri were overridded when python-tempestconf was executed. Users don't need to do that, python-tempestconf is able to read sourced credentials as explained in the note here [2]. 3. the uri address used was different from the one in overcloudrc file (it's better not to override credentials when python-tempestconf is able to read them, see point above)
Created attachment 1627354 [details] error_output I see now, the problem occurs when running tempest cleanup against undercloud. It seems that a user can't list floating ips with credentials from stackrc file which suggests that it might a problem of misconfigured user. (underclud) [stack@undercloud-0 cloud]$ source ../stackrc (undercloud) [stack@undercloud-0 cloud]$ openstack floating ip list NotFoundException: Unknown error (undercloud) [stack@undercloud-0 cloud]$ openstack floating ip list --debug <output omitted> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run result = self.run_subcommand(remainder) File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/display.py", line 119, in run self.produce_output(parsed_args, column_names, data) File "/usr/lib/python2.7/site-packages/cliff/lister.py", line 82, in produce_output parsed_args, File "/usr/lib/python2.7/site-packages/cliff/formatters/table.py", line 101, in emit_list self.add_rows(x, column_names, data) File "/usr/lib/python2.7/site-packages/cliff/formatters/table.py", line 80, in add_rows first_row = next(data_iter) File "/usr/lib/python2.7/site-packages/openstackclient/network/v2/floating_ip.py", line 399, in <genexpr> (utils.get_item_properties( File "/usr/lib/python2.7/site-packages/openstack/resource.py", line 898, in list exceptions.raise_from_response(response) File "/usr/lib/python2.7/site-packages/openstack/exceptions.py", line 205, in raise_from_response http_status=http_status, request_id=request_id NotFoundException: NotFoundException: Unknown error I'm also adding a full debug output as an attachment.
On a closer look, it looks like an issue in python-openstackclient. I believe when a user will be able to list floating ips on undercloud without the exception it will also fix the problem in tempest cleanup.
Tempest cleanup depends on the clients which list and delete resources. In this case, the floating ip client is failing due to the BZ #1765497. The BZ #1765497 got closed as WONTFIX so there is not much we can do in tempest cleanup about it. I have to close this as WONTFIX as well.