Bug 1760918

$ tempest cleanup --init-saved-state

2019-10-11 13:20:49.606 4593 INFO tempest.lib.common.rest_client [req-279f9f8b-070a-49d5-9fbb-9a85af8c8129 ] Request (main): 404 GET https://192.XXX.XX.X:XXXXX/v2.0/floatingips 0.239s
2019-10-11 13:20:49.607 4593 DEBUG tempest.lib.common.rest_client [req-279f9f8b-070a-49d5-9fbb-9a85af8c8129 ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'status': '404', u'content-length': '103', 'content-location': 'https://192.XXX.XX.X:XXXXX/v2.0/floatingips', u'date': 'Fri, 11 Oct 2019 17:20:49 GMT', u'content-type': 'application/json', u'connection': 'close', u'x-openstack-request-id': 'req-279f9f8b-070a-49d5-9fbb-9a85af8c8129'}
        Body: {"NeutronError": {"message": "The resource could not be found.", "type": "HTTPNotFound", "detail": ""}} _log_request_full /usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py:434
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup [-] Failure during cleanup: NotFound: Object not found
Details: {u'message': u'The resource could not be found.', u'type': u'HTTPNotFound', u'detail': u''}
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup Traceback (most recent call last):
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup   File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 110, in take_action
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup     self.init(parsed_args)
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup   File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 146, in init
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup     self._init_state()
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup   File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 332, in _init_state
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup     svc.run()
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup   File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup_service.py", line 134, in run
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup     self.save_state()
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup   File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup_service.py", line 452, in save_state
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup     flips = self.list()
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup   File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup_service.py", line 428, in list
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup     flips = client.list_floatingips(**self.tenant_filter)
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup   File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/floating_ips_client.py", line 65, in list_floatingips
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup     return self.list_resources(uri, **filters)
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup   File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/base.py", line 34, in list_resources
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup     resp, body = self.get(req_uri)
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup   File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 294, in get
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup     return self.request('GET', url, extra_headers, headers)
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup   File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup     self._error_checker(resp, resp_body)
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup   File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 774, in _error_checker
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup     raise exceptions.NotFound(resp_body, resp=resp)
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup NotFound: Object not found
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup Details: {u'message': u'The resource could not be found.', u'type': u'HTTPNotFound', u'detail': u''}
2019-10-11 13:20:49.607 4593 ERROR tempest.cmd.cleanup 
2019-10-11 13:20:49.609 4593 ERROR tempest [-] Object not found
Details: {u'message': u'The resource could not be found.', u'type': u'HTTPNotFound', u'detail': u''}: NotFound: Object not found
2019-10-11 13:20:49.609 4593 DEBUG tempest.cmd.main [-] tempest clean_up TempestCleanup clean_up /usr/lib/python2.7/site-packages/tempest/cmd/main.py:41
2019-10-11 13:20:49.610 4593 DEBUG tempest.cmd.main [-] tempest got an error: Object not found
Details: {u'message': u'The resource could not be found.', u'type': u'HTTPNotFound', u'detail': u''} clean_up /usr/lib/python2.7/site-packages/tempest/cmd/main.py:43

I reproduced the issue with tempest.conf which contained the following misconfigurations:

1. admin_password doesn't belong to [identity] section anymore [1], it should be under [auth] section.
2. I noticed that admin_password and uri were overridded when python-tempestconf was executed. Users don't need to do that, python-tempestconf is able to read sourced credentials as explained in the note here [2].
3. the uri address used was different from the one in overcloudrc file (it's better not to override credentials when python-tempestconf is able to read them, see point above)

Created attachment 1627354 [details]
error_output

I see now, the problem occurs when running tempest cleanup against undercloud. It seems that a user can't list floating ips with credentials from stackrc file which suggests that it might a problem of misconfigured user. 

(underclud) [stack@undercloud-0 cloud]$ source ../stackrc
(undercloud) [stack@undercloud-0 cloud]$ openstack floating ip list
NotFoundException: Unknown error
(undercloud) [stack@undercloud-0 cloud]$ openstack floating ip list --debug
<output omitted>
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/display.py", line 119, in run
    self.produce_output(parsed_args, column_names, data)
  File "/usr/lib/python2.7/site-packages/cliff/lister.py", line 82, in produce_output
    parsed_args,
  File "/usr/lib/python2.7/site-packages/cliff/formatters/table.py", line 101, in emit_list
    self.add_rows(x, column_names, data)
  File "/usr/lib/python2.7/site-packages/cliff/formatters/table.py", line 80, in add_rows
    first_row = next(data_iter)
  File "/usr/lib/python2.7/site-packages/openstackclient/network/v2/floating_ip.py", line 399, in <genexpr>
    (utils.get_item_properties(
  File "/usr/lib/python2.7/site-packages/openstack/resource.py", line 898, in list
    exceptions.raise_from_response(response)
  File "/usr/lib/python2.7/site-packages/openstack/exceptions.py", line 205, in raise_from_response
    http_status=http_status, request_id=request_id
NotFoundException: NotFoundException: Unknown error


I'm also adding a full debug output as an attachment.

On a closer look, it looks like an issue in python-openstackclient. I believe when a user will be able to list floating ips on undercloud without the exception it will also fix the problem in tempest cleanup.

Tempest cleanup depends on the clients which list and delete resources. In this case, the floating ip client is failing due to the BZ #1765497.
The BZ #1765497 got closed as WONTFIX so there is not much we can do in tempest cleanup about it. I have to close this as WONTFIX as well.