Bug 1479758 - [OSP12] tempest.api.compute.floating_ips.test_floating_ips_actions.FloatingIPsTestJSON and test_list_floating_ips.FloatingIPDetailsTestJSON was failed due to "Floating IP pool not found" [NEEDINFO]
[OSP12] tempest.api.compute.floating_ips.test_floating_ips_actions.FloatingIP...
Status: ASSIGNED
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tempest (Show other bugs)
12.0 (Pike)
x86_64 Linux
medium Severity low
: rc
: 12.0 (Pike)
Assigned To: Artom Lifshitz
Martin Kopec
: Reopened, Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-09 06:55 EDT by Artem Hrechanychenko
Modified: 2017-09-13 16:06 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-31 12:19:24 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
alifshit: needinfo? (ahrechan)


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 503797 None None None 2017-09-13 16:06 EDT

  None (edit)
Description Artem Hrechanychenko 2017-08-09 06:55:26 EDT
OSP12 HA+OC_SSL+UC_SSL
3ctrl+2comp nodes
puddle - OpenStack-12.0-RHEL-7-20170808.4

Description of problem:

tempest.api.compute.floating_ips.test_floating_ips_actions.FloatingIPsTestJSON
tempest.api.compute.floating_ips.test_list_floating_ips.FloatingIPDetailsTestJSON

<failure type="testtools.testresult.real._StringException">Traceback (most recent call last):
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/test.py", line 251, in setUpClass
    six.reraise(etype, value, trace)
  File "/usr/lib/python2.7/site-packages/tempest/test.py", line 244, in setUpClass
    cls.resource_setup()
  File "/usr/lib/python2.7/site-packages/tempest/api/compute/floating_ips/test_floating_ips_actions.py", line 53, in resource_setup
    pool=CONF.network.floating_network_name)['floating_ip']
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/compute/floating_ips_client.py", line 65, in create_floating_ip
    resp, body = self.post(url, post_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 270, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/compute/base_compute_client.py", line 48, in request
    method, url, extra_headers, headers, body, chunked)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 659, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 765, in _error_checker
    raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: {u'message': u'Floating IP pool not found.', u'code': 404}


Version-Release number of selected component (if applicable):

Version-Release number of selected component (if applicable):
openstack-swift-container-2.15.1-0.20170803052833.0b22193.el7ost.noarch
openstack-nova-common-16.0.0-0.20170805120344.5971dde.el7ost.noarch
openstack-ironic-conductor-8.0.1-0.20170805084044.c08c21e.el7ost.noarch
openstack-tempest-16.1.1-0.20170805070640.112eeb1.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
openstack-tripleo-validations-7.2.1-0.20170728183725.1f60b6f.el7ost.noarch
openstack-swift-proxy-2.15.1-0.20170803052833.0b22193.el7ost.noarch
openstack-heat-api-cfn-9.0.0-0.20170804164328.d5f78c2.el7ost.noarch
openstack-ironic-inspector-5.1.1-0.20170801061021.b4391de.el7ost.noarch
openstack-tripleo-ui-7.2.1-0.20170805065336.ac9467f.el7ost.noarch
openstack-tripleo-image-elements-7.0.0-0.20170725091025.f3f06c7.el7ost.noarch
openstack-nova-compute-16.0.0-0.20170805120344.5971dde.el7ost.noarch
openstack-neutron-openvswitch-11.0.0-0.20170804190459.el7ost.noarch
puppet-openstack_extras-11.3.0-0.20170805105506.dae9508.el7ost.noarch
openstack-tripleo-common-containers-7.4.1-0.20170805212013.430242a.el7ost.noarch
openstack-mistral-api-5.0.0-0.20170804112907.abebc64.el7ost.noarch
python-openstackclient-3.12.0-0.20170728181821.f67ebce.el7ost.noarch
openstack-selinux-0.8.7-2.el7ost.noarch
openstack-nova-placement-api-16.0.0-0.20170805120344.5971dde.el7ost.noarch
openstack-tripleo-puppet-elements-7.0.0-0.20170803140906.4e7d35d.el7ost.noarch
openstack-neutron-common-11.0.0-0.20170804190459.el7ost.noarch
puppet-openstacklib-11.3.0-0.20170805105609.cd97f82.el7ost.noarch
python-openstacksdk-0.9.17-0.20170621195806.7946243.el7ost.noarch
openstack-tripleo-common-7.4.1-0.20170805212013.430242a.el7ost.noarch
openstack-swift-object-2.15.1-0.20170803052833.0b22193.el7ost.noarch
openstack-heat-common-9.0.0-0.20170804164328.d5f78c2.el7ost.noarch
openstack-ironic-api-8.0.1-0.20170805084044.c08c21e.el7ost.noarch
openstack-mistral-engine-5.0.0-0.20170804112907.abebc64.el7ost.noarch
openstack-nova-api-16.0.0-0.20170805120344.5971dde.el7ost.noarch
openstack-nova-conductor-16.0.0-0.20170805120344.5971dde.el7ost.noarch
openstack-keystone-12.0.0-0.20170805005310.9cbd6bc.el7ost.noarch
openstack-heat-api-9.0.0-0.20170804164328.d5f78c2.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170805163045.el7ost.noarch
openstack-mistral-executor-5.0.0-0.20170804112907.abebc64.el7ost.noarch
python-openstack-mistral-5.0.0-0.20170804112907.abebc64.el7ost.noarch
openstack-nova-scheduler-16.0.0-0.20170805120344.5971dde.el7ost.noarch
openstack-glance-15.0.0-0.20170805121150.94df7f8.el7ost.noarch
openstack-swift-account-2.15.1-0.20170803052833.0b22193.el7ost.noarch
openstack-neutron-ml2-11.0.0-0.20170804190459.el7ost.noarch
openstack-mistral-common-5.0.0-0.20170804112907.abebc64.el7ost.noarch
openstack-zaqar-5.0.0-0.20170803163737.088a08c.el7ost.noarch
openstack-neutron-11.0.0-0.20170804190459.el7ost.noarch
openstack-heat-engine-9.0.0-0.20170804164328.d5f78c2.el7ost.noarch
openstack-ironic-common-8.0.1-0.20170805084044.c08c21e.el7ost.noarch
Comment 2 Artom Lifshitz 2017-08-17 20:14:23 EDT
Hi Artem,

My instinct is that this is unlikely to be a nova bug.

One possibility to explore is a tempest misconfiguration. Specifically, the value of 'floating_network_name' in the [network] section of tempest.conf should match what's deployed in neutron/nova.

It could be tempest not handling a specific case that's gone unnoticed until now. It could also be a legit nova bug. Without the full nova and tempest debug logs, it's hard to tell. Any chance those logs are available and could be attached to this bugzilla?

Cheers!
Comment 3 Artem Hrechanychenko 2017-08-18 03:46:36 EDT
Hi!
As you see from[0] the full tempest suite was run. So problem not in misconfiguration of tempest.

[0] https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/RHOS/view/RHOS12/job/OT_container_netiso_osp12_HA_SSL_localregistry_full_tempest/36/testReport/
Comment 4 Artom Lifshitz 2017-08-23 19:45:39 EDT
A tempest configuration option that doesn't match the deployed cloud won't cause all tests to fail, only the ones that depend on that particular configuration option. So in this case, we're looking at floating_network_name, and it needs to match what's in the cloud. Could you attach the tempest.conf used for the test run, and the output of 'openstack network list'?

Thanks!
Comment 5 Artom Lifshitz 2017-08-31 12:19:24 EDT
I'm going to close this bug for now to get it off the Compute DFG's pre-triage list. This is not a definitive action by any means, I encourage you to re-open this bug with the tempest.conf config file and the output of 'openstack network list' - or any other new information!

Cheers!
Comment 6 Artem Hrechanychenko 2017-09-07 07:29:36 EDT
ist

(overcloud) [stack@undercloud-0 ~]$ openstack network list


(overcloud) [stack@undercloud-0 ~]$

(overcloud) [stack@undercloud-0 ~]$ cat tempest-dir/etc/tempest.conf
[DEFAULT]
debug = true
use_stderr = false
log_file = tempest.log

[auth]
tempest_roles = _member_
admin_username = admin
admin_project_name = admin
admin_domain_name = Default

[compute]
image_ssh_user = cirros
flavor_ref = ac795af2-b70f-4cee-9ab4-93178fe3f5dd
flavor_ref_alt = b282dc71-e33b-4ea4-a0d9-27f8c000b032
image_ref = 8e5f8b4b-de02-49cb-a7cd-c2f3b35a6e72
image_ref_alt = 960dd483-4226-4f33-863e-59c507be0de7

[identity]
username = demo
tenant_name = demo
password = secrete
alt_username = alt_demo
alt_tenant_name = alt_demo
alt_password = secrete
admin_username = admin
admin_tenant_name = admin
admin_domain_name = Default
disable_ssl_certificate_validation = true
admin_password = Rexq2MTHsGzmP4jtZnNGtEgBP
uri = https://10.0.0.101:13000/v2.0
region = regionOne
uri_v3 = https://10.0.0.101:13000/v3
admin_tenant_id = ba2d711625a84ca9b7b0b2349786c46a

[object-storage]
operator_role = SwiftOperator

[data-processing]

[orchestration]
stack_owner_role = heat_stack_owner

[scenario]
img_dir = /home/stack/tempest-dir/etc

[oslo_concurrency]
lock_path = /tmp

[volume-feature-enabled]
bootable = true
volume_services = true
api_v2 = True
api_v3 = True
api_extensions = 
backup = False

[compute-feature-enabled]
live_migration = false
live_migrate_paused_instances = true
preserve_ports = true
api_extensions = NMN,OS-DCF,OS-EXT-AZ,OS-EXT-IMG-SIZE,OS-EXT-IPS,OS-EXT-IPS-MAC,OS-EXT-SRV-ATTR,OS-EXT-STS,OS-FLV-DISABLED,OS-FLV-EXT-DATA,OS-SCH-HNT,OS-SRV-USG,os-access-ips,os-admin-actions,os-admin-password,os-agents,os-aggregates,os-assisted-volume-snapshots,os-attach-interfaces,os-availability-zone,os-baremetal-ext-status,os-baremetal-nodes,os-block-device-mapping,os-block-device-mapping-v2-boot,os-cell-capacities,os-cells,os-certificates,os-cloudpipe,os-cloudpipe-update,os-config-drive,os-console-auth-tokens,os-console-output,os-consoles,os-create-backup,os-create-server-ext,os-deferred-delete,os-evacuate,os-extended-evacuate-find-host,os-extended-floating-ips,os-extended-hypervisors,os-extended-networks,os-extended-quotas,os-extended-rescue-with-image,os-extended-services,os-extended-services-delete,os-extended-status,os-extended-volumes,os-fixed-ips,os-flavor-access,os-flavor-extra-specs,os-flavor-manage,os-flavor-rxtx,os-flavor-swap,os-floating-ip-dns,os-floating-ip-pools,os-floating-ips,os-floating-ips-bulk,os-fping,os-hide-server-addresses,os-hosts,os-hypervisor-status,os-hypervisors,os-instance-actions,os-instance_usage_audit_log,os-keypairs,os-lock-server,os-migrate-server,os-migrations,os-multiple-create,os-networks,os-networks-associate,os-pause-server,os-personality,os-preserve-ephemeral-rebuild,os-quota-class-sets,os-quota-sets,os-rescue,os-security-group-default-rules,os-security-groups,os-server-diagnostics,os-server-external-events,os-server-group-quotas,os-server-groups,os-server-list-multi-status,os-server-password,os-server-sort-keys,os-server-start-stop,os-services,os-shelve,os-simple-tenant-usage,os-suspend-server,os-tenant-networks,os-used-limits,os-used-limits-for-admin,os-user-data,os-user-quotas,os-virtual-interfaces,os-volume-attachment-update,os-volumes

[network-feature-enabled]
ipv6_subnet_attributes = true
api_extensions = default-subnetpools,qos,network-ip-availability,network_availability_zone,auto-allocated-topology,ext-gw-mode,binding,agent,subnet_allocation,l3_agent_scheduler,tag,external-net,standard-attr-tag,flavors,net-mtu,availability_zone,qos-default,quotas,revision-if-match,l3-ha,provider,multi-provider,quota_details,address-scope,trunk,extraroute,net-mtu-writable,subnet-service-types,standard-attr-timestamp,service-type,qos-rule-type-details,l3-flavors,port-security,extra_dhcp_opt,standard-attr-revisions,pagination,sorting,security-group,dhcp_agent_scheduler,router_availability_zone,rbac-policies,qos-bw-limit-direction,tag-ext,standard-attr-description,router,allowed-address-pairs,project-id,trunk-details

[validation]
image_ssh_user = cirros

[service_available]
glance = True
manila = False
cinder = True
swift = True
sahara = False
panko = True
nova = True
neutron = True
trove = False
ceilometer = True
ironic = False
heat = True
zaqar = False
gnocchi = True
aodh = True
aodh_plugin = True
horizon = False

[image-feature-enabled]
api_v1 = False
api_v2 = True

[identity-feature-enabled]
api_v2 = True
api_v3 = True
api_extensions = https:,OS-REVOKE,OS-FEDERATION,OS-KSCRUD,OS-SIMPLE-CERT,OS-OAUTH1,OS-EC2
forbid_global_implied_dsr = True

[object-storage-feature-enabled]
discoverable_apis = formpost,bulk_delete,versioned_writes,container_quotas,tempurl,bulk_upload,ratelimit,slo,account_quotas,staticweb

[dashboard]
dashboard_url = https://10.0.0.101/dashboard/
login_url = https://10.0.0.101/dashboard/auth/login/
Comment 7 Artom Lifshitz 2017-09-08 09:59:03 EDT
The 404 error mislead me into believing that the floating IP pool was not found, when in actuality it's the entire os-floating-ips API endpoint that's not found, because it was deprecated in nova in API version 2.36 [1] [2]. So I think this is a legit bug, although in tempest and not in nova. I'm also not sure why it's not been a problem in upstream tempest CI, so I need to investigate further.

[1] https://developer.openstack.org/api-ref/compute/#floating-ips-os-floating-ips-deprecated
[2] https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id33
Comment 8 Artom Lifshitz 2017-09-13 16:03:23 EDT
Let's look at a random recent CI run in upstream tempest [1]. It's able to use the /os-floating-ips endpoint because it's not requesting any specific microversion, making nova default to 2.1, in which /os-floating-ips still exists.

I'm not sure what's different in our downstream CI runs. I'd like to take a look at the full tempest debug log, as well as the debug log from nova-api, and the nova.conf config file.

In the meantime, I'll still propose a patch upstream to cap the compute floating IPS tests to microversion 2.36.

[1] http://logs.openstack.org/89/502289/3/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/a166eb8/logs/tempest.txt.gz#_2017-09-11_06_34_46_318

Note You need to log in before you can comment on or make changes to this bug.