Bug 1272289

Summary: rdo-manager tempest smoke test failing on "floating ip pool not found'
Product: [Community] RDO Reporter: wes hayutin <whayutin>
Component: tempestAssignee: David Kranz <dkranz>
Status: CLOSED EOL QA Contact: tkammer
Severity: medium Docs Contact:
Priority: medium    
Version: LibertyCC: chrisw, ebagdasa, ihrachys, srevivo, tiswanso, whayutin, ymorkovn
Target Milestone: ---Keywords: Reopened
Target Release: Kilo   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-19 15:43:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
tempest log none

Description wes hayutin 2015-10-16 01:27:13 UTC 
Description of problem:

see job logs and tempest results for details.

https://ci.centos.org/view/rdo/job/rdo_manager-periodic-7-rdo-liberty-delorean_mgt-centos-7.0-templates-virthost-minimal-neutron-ml2-vxlan-smoke/56/testReport/tempest.scenario.test_volume_boot_pattern/TestVolumeBootPatternV2/test_volume_boot_pattern_compute_id_557cd2c2_4eb8_4dce_98be_f86765ff311b_image_smoke_volume_/

Traceback (most recent call last):
  File "/home/stack/tempest/tempest/test.py", line 127, in wrapper
    return f(self, *func_args, **func_kwargs)
  File "/home/stack/tempest/tempest/scenario/test_volume_boot_pattern.py", line 139, in test_volume_boot_pattern
    keypair)
  File "/home/stack/tempest/tempest/scenario/test_volume_boot_pattern.py", line 101, in _ssh_to_server
    ip = self.create_floating_ip(server)['ip']
  File "/home/stack/tempest/tempest/scenario/manager.py", line 552, in create_floating_ip
    floating_ip = (self.floating_ips_client.create_floating_ip(pool_name)
  File "/home/stack/tempest/tempest/services/compute/json/floating_ips_client.py", line 50, in create_floating_ip
    resp, body = self.post(url, post_body)
  File "/usr/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 259, in post
    return self.request('POST', url, extra_headers, headers, body)
  File "/usr/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 639, in request
    resp, resp_body)
  File "/usr/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 692, in _error_checker
    raise exceptions.NotFound(resp_body, resp=resp)
tempest_lib.exceptions.NotFound: Object not found
Details: {u'message': u'Floating ip pool not found.', u'code': 404}
Comment 1 Ihar Hrachyshka 2015-10-19 09:12:13 UTC 
The artifacts collected for the job run do not have debug logs enabled neither for nova nor for neutron, so it's not possible to debug what went wrong there.

Please attach the appropriate debug logs once it repeats.
Comment 2 Ihar Hrachyshka 2015-11-10 15:10:22 UTC 
OK, I am closing the bug since there are no logs that I could check for the culprit, and no activity from the reporter.
Comment 3 Evgeny 2015-11-12 20:20:35 UTC 
Hey Ihar,

I see basically the same behavior, however I think this is mostly a bug in tempest.

The problem is I don't have nova-network, therefore I don't need floating pool at all.

I see that contents of the file test_list_floating_ips.py are not identical with upstream and it may specifically line #7. I can't give you detailed commit.


Thanks,
Eugene
Comment 4 David Kranz 2015-11-13 15:15:53 UTC 
This error generally occurs when the default floating pool specified in nova.conf does not match any neutron external network. The nova apis for floating ips are supposed to work whether nova-network is being used or not. So if your deployment has floating ips through neutron you need to fix nova.conf. 

If you are saying that your deployment does not support floating ips at all, even with neutron, then the problem is that tempest does not support that use case very well. You would have to set use_floatingip_for_ssh to false in the [compute] section of tempest.conf but I believe many other tests will fail if there are no floating ips at all.
Comment 5 Timothy Swanson 2015-11-13 15:38:16 UTC 
Thanks for the info, David.  I work with Eugene and we'll try to figure out how to make the configs consistent.  We're deploying our overcloud with OSP director so we'll have to figure out how to manage the related nova.conf sections via corresponding tripleo-heat-template resources and params.
Comment 6 Timothy Swanson 2015-11-16 21:57:20 UTC 
I can get past this error via 
  1) set the following /etc/nova/nova.conf setting on the controllers:  
      default_floating_pool=<name of my external net>

  2) restart nova services 
       systemctl restart openstack-nova-api
       systemctl restart openstack-nova-conductor
       systemctl restart openstack-nova-scheduler

  3) rerun tempest

The problem with this work-around is it can't be orchestrated on overcloud deploy via openstack-tripleo-heat settings because puppet-nova doesn't have a way to set "default_floating_pool" in OSP7-- the patch to allow it was recently added to puppet-nova master via:  
           https://github.com/openstack/puppet-nova/commit/83ea9464a5c99733521541095104fa34d6ee2614

This looks like a tempest problem with assuming that the nova.conf default_floating_pool is always set.


NOTES:
  - Prereq: create external net and subnet with --disable_dhcp and --allocation-pool 
  - The name of my external net shows up under "nova floating-ip-pool-list"
  - I can create new floating IPs via:  nova floating-ip-create <name of external net>
      (same behavior as 'neutron floatingip-create <name of external net>')
Comment 7 David Kranz 2015-11-18 15:56:38 UTC 
I don't think this is a problem with tempest. Tempest is not assuming anything about nova.conf, but is assuming that the REST version of the cli command 'nova floating-ip-create' (with no pool argument) is working. Of course that implicitly means that nova.conf must be correct. Tempest will continue to test this api until/if it is deprecated. So IMO the bug here is with the deployment mechanism which you indicate has been fixed.
Comment 8 Evgeny 2015-11-24 17:27:23 UTC 
Can be closed now! Thanks for help!
Comment 9 Timothy Swanson 2015-11-24 17:57:56 UTC 
IMO, the reason this is a deficiency (bug) for tempest is that this assumption is in an API used in the general test setup methods for most (all?) tests requiring floating IPs.  This nova API has moved on and configuring "default_floating_pool" is not required for a "correct" deployment... It's fine if there's a specific test for the old API which requires the "default_floating_pool" nova.conf setting but it's actually blocking a ton of other validation when it's failing in the setup method.

Functionally, nova.conf "default_floating_pool" is not required so dependence on it should not be part of tempest test setup methods.
Comment 10 Daniel Mellado 2016-03-10 07:33:12 UTC 
*** Bug 1316294 has been marked as a duplicate of this bug. ***
Comment 11 wes hayutin 2016-03-10 12:49:48 UTC 
Created attachment 1134885 [details]
tempest log
Comment 12 wes hayutin 2016-03-10 12:55:28 UTC 
I've confirmed floating ips and pools are working on the deployment.  Maybe someone from the tempest crew can try this out to help get to the root cause.

Thanks
Comment 13 Chandan Kumar 2016-05-19 15:43:09 UTC 
This bug is against a Version which has reached End of Life.
If it's still present in supported release (http://releases.openstack.org), please update Version and reopen.