Bug 1314094 - unable to launch VM with rhosp 8 beta 7 with error resource not found
unable to launch VM with rhosp 8 beta 7 with error resource not found
Status: CLOSED NOTABUG
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
8.0 (Liberty)
Unspecified Unspecified
unspecified Severity medium
: ---
: 8.0 (Liberty)
Assigned To: Eoghan Glynn
nlevinki
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-02 17:22 EST by bigswitch
Modified: 2016-03-11 09:18 EST (History)
17 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-11 09:18:03 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
keystone log from controller (16.21 KB, text/plain)
2016-03-03 12:55 EST, bigswitch
no flags Details
nova-api log from controller (11.83 KB, text/plain)
2016-03-03 12:55 EST, bigswitch
no flags Details
nova-compute from compute node (24.73 KB, text/plain)
2016-03-03 12:55 EST, bigswitch
no flags Details
sosreport from controller-1 (16.77 MB, application/x-xz)
2016-03-08 06:39 EST, Ralf Boernemeier
no flags Details

  None (edit)
Description bigswitch 2016-03-02 17:22:07 EST
Description of problem:
Seen with rhosp 8 beta 7 with bigswitch ivs plugin. When attempting to launch an instance, it failed with the below error from nova-api log
This is one a single controller with two compute node setup

2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions [req-69531531-ed2a-4383-a5d8-3640a42b69fb 1696678afe154560811adf85ae5ae5e1 5debda3c410746ef8eb0b98c736549d9 - - -] Unexpected exception in API method
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 611, in create
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions     **create_kwargs)
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 149, in inner
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1581, in create
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1181, in _create_instance
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions     auto_disk_config, reservation_id, max_count)
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 955, in _validate_and_build_base_options
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions     pci_request_info, requested_networks)
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1059, in create_pci_requests_for_sriov_ports
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions     neutron = get_client(context, admin=True)
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 237, in get_client
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions     auth_token = _ADMIN_AUTH.get_token(_SESSION)
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions     return self.get_access(session).auth_token
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions     self.auth_ref = self.get_auth_ref(session)
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py", line 88, in get_auth_ref
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions     authenticated=False, log=False)
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions     return self.request(url, 'POST', **kwargs)
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions     raise exceptions.from_response(resp, method, url)
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions NotFound: The resource could not be found. (HTTP 404)
2016-03-02 17:20:17.429 11271 ERROR nova.api.openstack.extensions


Version-Release number of selected component (if applicable):
rhosp 8 beta 7

How reproducible:
100%

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 2 Kashyap Chamarthy 2016-03-03 09:43:28 EST
I'm afraid, the bug report doesn't contain sufficient info to 
investigate.

Please provide:

 - A rough outline of the reproducer.  Or even better: exactly how you 
   launched the instance (a command-line invocation of Nova is
   preferred)

 - Any environment details

 - Sanitized Keystone configuration (from Controller)

 - Sanitized Nova configuration files (from Controller & Compute)

 - Keystone logs (from Controller node)

 - Nova logs (from Controller/Compute node)

That said, the error seems to be originating from Keystone (maybe a 
bogus user ID or token).  Could be any reason.  Probably the logs will
shed more light on this.


From your description, you say you're using Big Switch IVS plugin.  I'd 
assume it's this one --
http://docs.openstack.org/liberty/config-reference/content/networking-plugin-bigswitch.html
Comment 3 bigswitch 2016-03-03 12:53:32 EST
Hi
The instance is launched from Horizon, 
1) in admin tenant, create a new project P1, and user user1 with role admin and assigned to project P1
2) logout admin user, and login as user1
3) create a network, router, and attached network to router
4) launch a VM using cirros image

keystone config file:
[DEFAULT]
admin_token = 6wfzFyqWtvUksBaX8DZKX4ngt
notification_format = basic
debug = False
verbose = False
log_dir = /var/log/keystone
use_syslog = False
syslog_log_facility = LOG_USER
use_stderr = True
notification_driver =messaging
public_port=5000
admin_bind_host=192.168.0.24
public_bind_host=172.16.2.6
admin_port=35357
[assignment]
[auth]
[cache]
[catalog]
template_file = /etc/keystone/default_catalog.templates
driver = keystone.catalog.backends.sql.Catalog
[cors]
[cors.subdomain]
[credential]
[database]
connection = mysql://keystone:6wfzFyqWtvUksBaX8DZKX4ngt@172.16.2.4/keystone
idle_timeout = 3600
min_pool_size = 1
max_pool_size = 10
max_retries = 10
retry_interval = 10
max_overflow = 20
[domain_config]
[endpoint_filter]
[endpoint_policy]
[eventlet_server]
public_workers = 4
admin_workers = 4
[eventlet_server_ssl]
[federation]
[fernet_tokens]
key_repository = /etc/keystone/fernet-keys
[identity]
[identity_mapping]
[kvs]
[ldap]
[matchmaker_redis]
[matchmaker_ring]
[memcache]
[oauth1]
[os_inherit]
[oslo_messaging_amqp]
[oslo_messaging_qpid]
[oslo_messaging_rabbit]
rabbit_hosts = 172.16.2.6
rabbit_use_ssl = False
rabbit_userid = guest
rabbit_password = guest
rabbit_virtual_host = /
rabbit_ha_queues = True
heartbeat_timeout_threshold = 60
heartbeat_rate = 2
[oslo_middleware]
[oslo_policy]
[paste_deploy]
config_file = /usr/share/keystone/keystone-dist-paste.ini
[policy]
[resource]
[revoke]
driver = keystone.contrib.revoke.backends.sql.Revoke
[role]
[saml]
[signing]
certfile = /etc/keystone/ssl/certs/signing_cert.pem
keyfile = /etc/keystone/ssl/private/signing_key.pem
ca_certs = /etc/keystone/ssl/certs/ca.pem
ca_key = /etc/keystone/ssl/private/cakey.pem
key_size = 2048
cert_subject = /C=US/ST=Unset/L=Unset/O=Unset/CN=www.example.com
[ssl]
enable=False
[token]
expiration = 3600
provider = keystone.token.providers.uuid.Provider
driver = keystone.token.persistence.backends.sql.Token
revoke_by_id = True
[tokenless_auth]
[trust]
[ec2]
driver=keystone.contrib.ec2.backends.sql.Ec2

Controller nova.conf:

[DEFAULT]
novncproxy_host=172.16.2.6
novncproxy_port=6080
instance_name_template=instance-%08x
use_ipv6=False
notify_on_state_change=vm_and_task_state
notify_api_faults=False
state_path=/var/lib/nova
report_interval=10
enabled_apis=ec2,osapi_compute,metadata
ec2_listen=172.16.2.6
ec2_listen_port=8773
ec2_workers=4
osapi_compute_listen=172.16.2.6
osapi_compute_listen_port=8774
osapi_compute_workers=4
metadata_listen=172.16.2.6
metadata_listen_port=8775
metadata_workers=4
service_down_time=60
rootwrap_config=/etc/nova/rootwrap.conf
volume_api_class=nova.volume.cinder.API
api_paste_config=api-paste.ini
auth_strategy=keystone
use_forwarded_for=False
fping_path=/usr/sbin/fping
cpu_allocation_ratio=16.0
ram_allocation_ratio=1.0
network_api_class=nova.network.neutronv2.api.API
default_floating_pool=public
dhcp_domain=
security_group_api=neutron
memcached_servers=172.16.2.6:11211
disk_allocation_ratio=1.0
max_io_ops_per_host=8
max_instances_per_host=50
scheduler_host_manager=nova.scheduler.host_manager.HostManager
scheduler_host_subset_size=1
scheduler_available_filters=nova.scheduler.filters.all_filters
scheduler_weight_classes=nova.scheduler.weights.all_weighers
scheduler_use_baremetal_filters=False
scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
scheduler_max_attempts=3
vif_plugging_is_fatal=True
vif_plugging_timeout=300
firewall_driver=nova.virt.firewall.NoopFirewallDriver
debug=False
verbose=False
log_dir=/var/log/nova
use_syslog=False
syslog_log_facility=LOG_USER
use_stderr=True
notification_topics=notifications
rpc_backend=rabbit
image_service=nova.image.glance.GlanceImageService
lock_path=/var/lib/nova/tmp
osapi_volume_listen=172.16.2.6
novncproxy_base_url=http://172.16.2.6:6080/vnc_auto.html
[api_database]
[barbican]
[cells]
[cinder]
catalog_info=volumev2:cinderv2:internalURL
[conductor]
use_local=False
[cors]
[cors.subdomain]
[database]
connection=mysql://nova:Vt7PPdyMVtCsNFUmzb9ztZ3cV@172.16.2.4/nova
idle_timeout=3600
min_pool_size=1
max_pool_size=10
max_retries=10
retry_interval=10
max_overflow=20
[ephemeral_storage_encryption]
[glance]
api_servers=http://172.16.1.4:9292
[guestfs]
[hyperv]
[image_file_url]
[ironic]
[keymgr]
[keystone_authtoken]
auth_uri=http://172.16.2.4:5000/v2.0
identity_uri=http://192.168.0.22:35357
admin_user=nova
admin_password=Vt7PPdyMVtCsNFUmzb9ztZ3cV
admin_tenant_name=service
[libvirt]
[matchmaker_redis]
[matchmaker_ring]
[metrics]
[neutron]
service_metadata_proxy=True
metadata_proxy_shared_secret =GG8PhzrZfGrgNHZ8RGHCm83AP
url=http://172.16.2.4:9696
admin_username=neutron
admin_password=pwwaAtn44xdewfTtgfRkVQ4EX
admin_tenant_name=service
region_name=RegionOne
admin_auth_url=http://192.168.0.22:35357
auth_strategy=keystone
ovs_bridge=br-int
extension_sync_interval=600
timeout=30
default_tenant_id=default
[osapi_v21]
[oslo_concurrency]
[oslo_messaging_amqp]
[oslo_messaging_qpid]
[oslo_messaging_rabbit]
amqp_durable_queues=False
kombu_reconnect_delay=1.0
rabbit_hosts=172.16.2.6
rabbit_use_ssl=False
rabbit_userid=guest
rabbit_password=guest
rabbit_virtual_host=/
rabbit_ha_queues=True
heartbeat_timeout_threshold=60
heartbeat_rate=2
[oslo_middleware]
[rdp]
[serial_console]
[spice]
[ssl]
[trusted_computing]
[upgrade_levels]
[vmware]
[vnc]
[workarounds]
[xenserver]
[zookeeper]
[osapi_v3]
enabled=True

Compute nova.conf

[DEFAULT]
internal_service_availability_zone=internal
default_availability_zone=nova
my_ip=172.16.2.7
use_ipv6=False
notify_on_state_change=vm_and_task_state
notify_api_faults=False
state_path=/var/lib/nova
report_interval=10
compute_manager=nova.compute.manager.ComputeManager
service_down_time=60
instance_usage_audit_period=hour
rootwrap_config=/etc/nova/rootwrap.conf
auth_strategy=keystone
allow_resize_to_same_host=False
instance_usage_audit=True
heal_instance_info_cache_interval=60
reserved_host_memory_mb=512
network_api_class=nova.network.neutronv2.api.API
linuxnet_interface_driver=nova.network.linux_net.LinuxOVSInterfaceDriver
dhcp_domain=
security_group_api=neutron
compute_driver=libvirt.LibvirtDriver
vif_plugging_is_fatal=True
vif_plugging_timeout=300
firewall_driver=nova.virt.firewall.NoopFirewallDriver
force_raw_images=True
debug=False
verbose=False
log_dir=/var/log/nova
use_syslog=False
syslog_log_facility=LOG_USER
use_stderr=True
notification_topics=notifications
rpc_backend=rabbit
vncserver_proxyclient_address=172.16.2.7
vnc_keymap=en-us
vnc_enabled=True
image_service=nova.image.glance.GlanceImageService
lock_path=/var/lib/nova/tmp
vncserver_listen=0.0.0.0
novncproxy_base_url=http://10.8.68.111:6080/vnc_auto.html
[api_database]
[barbican]
[cells]
[cinder]
catalog_info=volumev2:cinderv2:internalURL
[conductor]
[cors]
[cors.subdomain]
[database]
[ephemeral_storage_encryption]
[glance]
api_servers=http://172.16.1.4:9292
[guestfs]
[hyperv]
[image_file_url]
[ironic]
[keymgr]
[keystone_authtoken]
[libvirt]
virt_type=kvm
inject_password=False
inject_key=False
inject_partition=-2
cpu_mode=host-model
[matchmaker_redis]
[matchmaker_ring]
[metrics]
[neutron]
url=http://172.16.2.4:9696
admin_username=neutron
admin_password=pwwaAtn44xdewfTtgfRkVQ4EX
admin_tenant_name=service
region_name=RegionOne
admin_auth_url=http://192.168.0.22:35357/v2.0
auth_strategy=keystone
ovs_bridge=br-int
extension_sync_interval=600
timeout=30
default_tenant_id=default
[osapi_v21]
[oslo_concurrency]
[oslo_messaging_amqp]
[oslo_messaging_qpid]
[oslo_messaging_rabbit]
amqp_durable_queues=False
kombu_reconnect_delay=1.0
rabbit_hosts=172.16.2.6
rabbit_use_ssl=False
rabbit_userid=guest
rabbit_password=guest
rabbit_virtual_host=/
rabbit_ha_queues=True
heartbeat_timeout_threshold=60
heartbeat_rate=2
[oslo_middleware]
[rdp]
[serial_console]
[spice]
[ssl]
[trusted_computing]
[upgrade_levels]
[vmware]
[vnc]
[workarounds]
[xenserver]
[zookeeper]

The logs from nova and keystone, compute and controller is attached
Comment 4 bigswitch 2016-03-03 12:55 EST
Created attachment 1132877 [details]
keystone log from controller
Comment 5 bigswitch 2016-03-03 12:55 EST
Created attachment 1132878 [details]
nova-api log from controller
Comment 6 bigswitch 2016-03-03 12:55 EST
Created attachment 1132879 [details]
nova-compute from compute node
Comment 7 Kashyap Chamarthy 2016-03-04 10:51:48 EST
A couple more things:

  - Can you reproduce this issue _without_ BigSwitch plugin?  That will 
    allow us narrow down the issue (I don't have the hardware to try
    to reproduce this myself)

  - Also, you might want to provide the Neutron network configuration  
    details (including BigSwitch plugin specific configs) -- so 
    Networking experts might be able to look at.
Comment 8 bigswitch 2016-03-04 17:37:27 EST
Hi
I tried with vanilla installation, with linux bonding configured and still seeing the same issue

below is the log and neutron.conf and ml2_conf.ini file. Let me know if something else is needed


2016-03-04 17:04:37.026 2943 WARNING oslo_config.cfg [-] Option "novncproxy_base_url" from group "DEFAULT" is deprecated. Use option "novncproxy_base_url" from group "vnc".
2016-03-04 17:32:05.980 3541 WARNING oslo_config.cfg [req-b7d16a6a-9b18-4b08-99f9-0555a818469e 284a376ac56e4aa1a025a87d9fb8d4b4 1c0974c84f3f410d9dbac843db95b226 - - -] Option "auth_strategy" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
2016-03-04 17:32:05.981 3541 WARNING oslo_config.cfg [req-b7d16a6a-9b18-4b08-99f9-0555a818469e 284a376ac56e4aa1a025a87d9fb8d4b4 1c0974c84f3f410d9dbac843db95b226 - - -] Option "admin_auth_url" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
2016-03-04 17:32:05.981 3541 WARNING oslo_config.cfg [req-b7d16a6a-9b18-4b08-99f9-0555a818469e 284a376ac56e4aa1a025a87d9fb8d4b4 1c0974c84f3f410d9dbac843db95b226 - - -] Option "admin_username" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
2016-03-04 17:32:05.982 3541 WARNING oslo_config.cfg [req-b7d16a6a-9b18-4b08-99f9-0555a818469e 284a376ac56e4aa1a025a87d9fb8d4b4 1c0974c84f3f410d9dbac843db95b226 - - -] Option "admin_password" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
2016-03-04 17:32:05.982 3541 WARNING oslo_config.cfg [req-b7d16a6a-9b18-4b08-99f9-0555a818469e 284a376ac56e4aa1a025a87d9fb8d4b4 1c0974c84f3f410d9dbac843db95b226 - - -] Option "admin_tenant_name" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions [req-b7d16a6a-9b18-4b08-99f9-0555a818469e 284a376ac56e4aa1a025a87d9fb8d4b4 1c0974c84f3f410d9dbac843db95b226 - - -] Unexpected exception in API method
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 611, in create
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     **create_kwargs)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 149, in inner
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1581, in create
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1181, in _create_instance
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     auto_disk_config, reservation_id, max_count)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 955, in _validate_and_build_base_options
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     pci_request_info, requested_networks)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1059, in create_pci_requests_for_sriov_ports
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     neutron = get_client(context, admin=True)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 237, in get_client
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     auth_token = _ADMIN_AUTH.get_token(_SESSION)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     return self.get_access(session).auth_token
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     self.auth_ref = self.get_auth_ref(session)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py", line 88, in get_auth_ref
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     authenticated=False, log=False)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     return self.request(url, 'POST', **kwargs)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     raise exceptions.from_response(resp, method, url)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions NotFound: The resource could not be found. (HTTP 404)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions


neutron.conf
[DEFAULT]
verbose = False
router_distributed = False
debug = False
state_path = /var/lib/neutron
use_syslog = False
use_stderr = True
log_dir =/var/log/neutron
bind_host = 172.16.2.7
bind_port = 9696
core_plugin =ml2
service_plugins =router
auth_strategy = keystone
base_mac = fa:16:3e:00:00:00
mac_generation_retries = 16
dhcp_lease_duration = 86400
dhcp_agent_notification = True
allow_bulk = True
allow_pagination = False
allow_sorting = False
allow_overlapping_ips = True
advertise_mtu = False
agent_down_time = 75
router_scheduler_driver = neutron.scheduler.l3_agent_scheduler.ChanceScheduler
allow_automatic_l3agent_failover = False
dhcp_agents_per_network = 1
l3_ha = False
api_workers = 4
rpc_workers = 4
use_ssl = False
notify_nova_on_port_status_changes = True
notify_nova_on_port_data_changes = True
nova_url = http://172.16.2.5:8774/v2/
send_events_interval = 2
rpc_response_timeout=60
rpc_backend=rabbit
control_exchange=neutron
lock_path=/var/lib/neutron/lock
[matchmaker_redis]
[matchmaker_ring]
[quotas]
[agent]
root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf
report_interval = 30
[keystone_authtoken]
auth_uri = http://172.16.2.5:5000/v2.0
identity_uri = http://192.168.0.6:35357
admin_tenant_name = service
admin_user = neutron
admin_password = hZYQNjHTZj8ZNu4BT9WVMXhKr
[database]
connection = mysql://neutron:hZYQNjHTZj8ZNu4BT9WVMXhKr@172.16.2.5/ovs_neutron?charset=utf8
max_retries = 10
retry_interval = 10
min_pool_size = 1
max_pool_size = 10
idle_timeout = 3600
max_overflow = 20
[nova]
auth_plugin =password
project_name=service
username=nova
password=Pn8RDsP4nFTGWdWCXwRZwaJrY
project_domain_id=default
tenant_name=service
user_domain_id=default
auth_url=http://192.168.0.6:35357
[oslo_concurrency]
[oslo_policy]
[oslo_messaging_amqp]
[oslo_messaging_qpid]
[oslo_messaging_rabbit]
kombu_reconnect_delay = 1.0
rabbit_hosts = 172.16.2.7
rabbit_use_ssl = False
rabbit_userid = guest
rabbit_password = guest
rabbit_virtual_host = /
rabbit_ha_queues = True
heartbeat_rate=2
heartbeat_timeout_threshold=60
[qos]
notification_drivers = message_queue

ml2_conf.ini file
[ml2]
type_drivers = vxlan,vlan,flat,gre
tenant_network_types = vlan
mechanism_drivers =openvswitch
extension_drivers =
path_mtu = 0
[ml2_type_flat]
flat_networks =datacentre
[ml2_type_vlan]
network_vlan_ranges =datacentre:1001:2000
[ml2_type_gre]
tunnel_id_ranges =1:1000
[ml2_type_vxlan]
vni_ranges =1:1000
vxlan_group =224.0.0.1
[ml2_type_geneve]
[securitygroup]
enable_security_group = True
Comment 9 Ralf Boernemeier 2016-03-08 06:16:52 EST
Hi,

I'm running into the same problem with OSP 8 Beta 7 on 3x Controller + 3x Compute Node configuration *without* any extra plugins, using vxlan configuration.

Here is the command I have uesed to create the oovercloud:

openstack overcloud deploy --stack osp8r1 --templates ~/templates/my-overcloud -e ~/templates/my-overcloud/environments/network-isolation.yaml -e ~/templates/network-environment.yaml -e ~/templates/my-overcloud/rhel-registration/environment-rhel-registration.yaml -e ~/templates/my-overcloud/rhel-registration/rhel-registration-resource-registry.yaml  -e ~/templates/storage-environment.yaml  -e  ~/templates/pre-deployment.yaml -e ~/templates/post-deployment.yaml --control-scale 3 --compute-scale 3 --control-flavor control --compute-flavor compute --ntp-server time.pdb.fsc.net --neutron-network-type vxlan --neutron-tunnel-types vxlan

When I try to create an instance I saw the following error:

[stack@osp8dr1 ~(OC-admin)]$ nova boot --flavor fjsv.rhel_xl --boot-volume 934ef6b8-4bd8-4177-b262-9acc3143644f --key-name key-admin --nic net-id=d1db1bfd-005c-408b-bad5-9dcd32599d24 --security-group sec1 --availability-zone nova:osp8r1-novacompute-0.localdomain vmA1_1
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'keystoneclient.exceptions.NotFound'> (HTTP 500) (Request-ID: req-d5939278-7245-435b-9e16-3b425a1357f7)

In the API log from controller-1 I have the following traceback:

2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions [req-d5939278-7245-435b-9e16-3b425a1357f7 0d96230dfeeb46cd858141e0459ffee9 5e7f78267faa4832a292d65f220bca19 - - -] Unexpected exception in API method
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 611, in create
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     **create_kwargs)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 149, in inner
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1581, in create
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1181, in _create_instance
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     auto_disk_config, reservation_id, max_count)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 955, in _validate_and_build_base_options
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     pci_request_info, requested_networks)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1059, in create_pci_requests_for_sriov_ports
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     neutron = get_client(context, admin=True)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 237, in get_client
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     auth_token = _ADMIN_AUTH.get_token(_SESSION)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     return self.get_access(session).auth_token
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     self.auth_ref = self.get_auth_ref(session)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py", line 88, in get_auth_ref
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     authenticated=False, log=False)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     return self.request(url, 'POST', **kwargs)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     raise exceptions.from_response(resp, method, url)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions NotFound: The resource could not be found. (HTTP 404)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions

I will attach sosreport from all 3 Controller-nodes.

Regards, 

Ralf (Fujitsu)
Comment 10 Ralf Boernemeier 2016-03-08 06:39 EST
Created attachment 1134103 [details]
sosreport from controller-1
Comment 11 nlevinki 2016-03-10 03:48:25 EST
Hi,
I think this ticket is actually https://bugzilla.redhat.com/show_bug.cgi?id=1308422
There is a workaround until the fix will get in
see comment #4
Workaround: 
change the admin_auth_url to  http://192.0.2.18:35357/v2.0 on all controllers.
pcs resource restart openstack-nova-api-clone
Comment 12 Vladik Romanovsky 2016-03-10 11:51:43 EST
Hello,

Looking at both of the provided nova.conf files, it appears that you are using a wrong keystone endpoint. (Missing /v2.0 at the end of your admin_auth_url)

Please follow comment 11 for the workaround.

Thanks,
Vladik
Comment 13 bigswitch 2016-03-10 19:28:45 EST
Hi,
I am able to launch instance after following comment 11. Please close this ticket.


Thanks

Song
Comment 14 Kashyap Chamarthy 2016-03-11 09:18:03 EST
Closing this bug as per comments #11, #12, #13.  (Thanks Nathan & Vladik for confirming).  

Just re-stating the root cause:

Explicitly specifying the Keystone end point version as part of 'admin_auth_url' in nova.conf resolves this:

    admin_auth_url=http://$IP-ADDRESS:35357/v2.0

Note You need to log in before you can comment on or make changes to this bug.