Hi Artem,

My instinct is that this is unlikely to be a nova bug.

One possibility to explore is a tempest misconfiguration. Specifically, the value of 'floating_network_name' in the [network] section of tempest.conf should match what's deployed in neutron/nova.

It could be tempest not handling a specific case that's gone unnoticed until now. It could also be a legit nova bug. Without the full nova and tempest debug logs, it's hard to tell. Any chance those logs are available and could be attached to this bugzilla?

Cheers!

Hi!
As you see from[0] the full tempest suite was run. So problem not in misconfiguration of tempest.

[0] https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/RHOS/view/RHOS12/job/OT_container_netiso_osp12_HA_SSL_localregistry_full_tempest/36/testReport/

A tempest configuration option that doesn't match the deployed cloud won't cause all tests to fail, only the ones that depend on that particular configuration option. So in this case, we're looking at floating_network_name, and it needs to match what's in the cloud. Could you attach the tempest.conf used for the test run, and the output of 'openstack network list'?

Thanks!

I'm going to close this bug for now to get it off the Compute DFG's pre-triage list. This is not a definitive action by any means, I encourage you to re-open this bug with the tempest.conf config file and the output of 'openstack network list' - or any other new information!

Cheers!

ist

(overcloud) [stack@undercloud-0 ~]$ openstack network list


(overcloud) [stack@undercloud-0 ~]$

(overcloud) [stack@undercloud-0 ~]$ cat tempest-dir/etc/tempest.conf
[DEFAULT]
debug = true
use_stderr = false
log_file = tempest.log

[auth]
tempest_roles = _member_
admin_username = admin
admin_project_name = admin
admin_domain_name = Default

[compute]
image_ssh_user = cirros
flavor_ref = ac795af2-b70f-4cee-9ab4-93178fe3f5dd
flavor_ref_alt = b282dc71-e33b-4ea4-a0d9-27f8c000b032
image_ref = 8e5f8b4b-de02-49cb-a7cd-c2f3b35a6e72
image_ref_alt = 960dd483-4226-4f33-863e-59c507be0de7

[identity]
username = demo
tenant_name = demo
password = secrete
alt_username = alt_demo
alt_tenant_name = alt_demo
alt_password = secrete
admin_username = admin
admin_tenant_name = admin
admin_domain_name = Default
disable_ssl_certificate_validation = true
admin_password = Rexq2MTHsGzmP4jtZnNGtEgBP
uri = https://10.0.0.101:13000/v2.0
region = regionOne
uri_v3 = https://10.0.0.101:13000/v3
admin_tenant_id = ba2d711625a84ca9b7b0b2349786c46a

[object-storage]
operator_role = SwiftOperator

[data-processing]

[orchestration]
stack_owner_role = heat_stack_owner

[scenario]
img_dir = /home/stack/tempest-dir/etc

[oslo_concurrency]
lock_path = /tmp

[volume-feature-enabled]
bootable = true
volume_services = true
api_v2 = True
api_v3 = True
api_extensions = 
backup = False

[compute-feature-enabled]
live_migration = false
live_migrate_paused_instances = true
preserve_ports = true
api_extensions = NMN,OS-DCF,OS-EXT-AZ,OS-EXT-IMG-SIZE,OS-EXT-IPS,OS-EXT-IPS-MAC,OS-EXT-SRV-ATTR,OS-EXT-STS,OS-FLV-DISABLED,OS-FLV-EXT-DATA,OS-SCH-HNT,OS-SRV-USG,os-access-ips,os-admin-actions,os-admin-password,os-agents,os-aggregates,os-assisted-volume-snapshots,os-attach-interfaces,os-availability-zone,os-baremetal-ext-status,os-baremetal-nodes,os-block-device-mapping,os-block-device-mapping-v2-boot,os-cell-capacities,os-cells,os-certificates,os-cloudpipe,os-cloudpipe-update,os-config-drive,os-console-auth-tokens,os-console-output,os-consoles,os-create-backup,os-create-server-ext,os-deferred-delete,os-evacuate,os-extended-evacuate-find-host,os-extended-floating-ips,os-extended-hypervisors,os-extended-networks,os-extended-quotas,os-extended-rescue-with-image,os-extended-services,os-extended-services-delete,os-extended-status,os-extended-volumes,os-fixed-ips,os-flavor-access,os-flavor-extra-specs,os-flavor-manage,os-flavor-rxtx,os-flavor-swap,os-floating-ip-dns,os-floating-ip-pools,os-floating-ips,os-floating-ips-bulk,os-fping,os-hide-server-addresses,os-hosts,os-hypervisor-status,os-hypervisors,os-instance-actions,os-instance_usage_audit_log,os-keypairs,os-lock-server,os-migrate-server,os-migrations,os-multiple-create,os-networks,os-networks-associate,os-pause-server,os-personality,os-preserve-ephemeral-rebuild,os-quota-class-sets,os-quota-sets,os-rescue,os-security-group-default-rules,os-security-groups,os-server-diagnostics,os-server-external-events,os-server-group-quotas,os-server-groups,os-server-list-multi-status,os-server-password,os-server-sort-keys,os-server-start-stop,os-services,os-shelve,os-simple-tenant-usage,os-suspend-server,os-tenant-networks,os-used-limits,os-used-limits-for-admin,os-user-data,os-user-quotas,os-virtual-interfaces,os-volume-attachment-update,os-volumes

[network-feature-enabled]
ipv6_subnet_attributes = true
api_extensions = default-subnetpools,qos,network-ip-availability,network_availability_zone,auto-allocated-topology,ext-gw-mode,binding,agent,subnet_allocation,l3_agent_scheduler,tag,external-net,standard-attr-tag,flavors,net-mtu,availability_zone,qos-default,quotas,revision-if-match,l3-ha,provider,multi-provider,quota_details,address-scope,trunk,extraroute,net-mtu-writable,subnet-service-types,standard-attr-timestamp,service-type,qos-rule-type-details,l3-flavors,port-security,extra_dhcp_opt,standard-attr-revisions,pagination,sorting,security-group,dhcp_agent_scheduler,router_availability_zone,rbac-policies,qos-bw-limit-direction,tag-ext,standard-attr-description,router,allowed-address-pairs,project-id,trunk-details

[validation]
image_ssh_user = cirros

[service_available]
glance = True
manila = False
cinder = True
swift = True
sahara = False
panko = True
nova = True
neutron = True
trove = False
ceilometer = True
ironic = False
heat = True
zaqar = False
gnocchi = True
aodh = True
aodh_plugin = True
horizon = False

[image-feature-enabled]
api_v1 = False
api_v2 = True

[identity-feature-enabled]
api_v2 = True
api_v3 = True
api_extensions = https:,OS-REVOKE,OS-FEDERATION,OS-KSCRUD,OS-SIMPLE-CERT,OS-OAUTH1,OS-EC2
forbid_global_implied_dsr = True

[object-storage-feature-enabled]
discoverable_apis = formpost,bulk_delete,versioned_writes,container_quotas,tempurl,bulk_upload,ratelimit,slo,account_quotas,staticweb

[dashboard]
dashboard_url = https://10.0.0.101/dashboard/
login_url = https://10.0.0.101/dashboard/auth/login/

The 404 error mislead me into believing that the floating IP pool was not found, when in actuality it's the entire os-floating-ips API endpoint that's not found, because it was deprecated in nova in API version 2.36 [1] [2]. So I think this is a legit bug, although in tempest and not in nova. I'm also not sure why it's not been a problem in upstream tempest CI, so I need to investigate further.

[1] https://developer.openstack.org/api-ref/compute/#floating-ips-os-floating-ips-deprecated
[2] https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id33

Let's look at a random recent CI run in upstream tempest [1]. It's able to use the /os-floating-ips endpoint because it's not requesting any specific microversion, making nova default to 2.1, in which /os-floating-ips still exists.

I'm not sure what's different in our downstream CI runs. I'd like to take a look at the full tempest debug log, as well as the debug log from nova-api, and the nova.conf config file.

In the meantime, I'll still propose a patch upstream to cap the compute floating IPS tests to microversion 2.36.

[1] http://logs.openstack.org/89/502289/3/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/a166eb8/logs/tempest.txt.gz#_2017-09-11_06_34_46_318

I'll try to update tempet.conf with floating ip pool and re-run this test

The upstream tempest change [1] has just merged, it should prevent floating IPs tests from being run on microversions that don't support them. That doesn't really answer the question of why those tests were fine in upstream CI but failed to pass downstream, but it should prevent future downstream failures.

[1] https://review.openstack.org/#/c/503797/

Upstream patch has been merged and test execution is now successful using the latest microversion.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462