I'm afraid, the bug report doesn't contain sufficient info to 
investigate.

Please provide:

 - A rough outline of the reproducer.  Or even better: exactly how you 
   launched the instance (a command-line invocation of Nova is
   preferred)

 - Any environment details

 - Sanitized Keystone configuration (from Controller)

 - Sanitized Nova configuration files (from Controller & Compute)

 - Keystone logs (from Controller node)

 - Nova logs (from Controller/Compute node)

That said, the error seems to be originating from Keystone (maybe a 
bogus user ID or token).  Could be any reason.  Probably the logs will
shed more light on this.


From your description, you say you're using Big Switch IVS plugin.  I'd 
assume it's this one --
http://docs.openstack.org/liberty/config-reference/content/networking-plugin-bigswitch.html

Hi
The instance is launched from Horizon, 
1) in admin tenant, create a new project P1, and user user1 with role admin and assigned to project P1
2) logout admin user, and login as user1
3) create a network, router, and attached network to router
4) launch a VM using cirros image

keystone config file:
[DEFAULT]
admin_token = 6wfzFyqWtvUksBaX8DZKX4ngt
notification_format = basic
debug = False
verbose = False
log_dir = /var/log/keystone
use_syslog = False
syslog_log_facility = LOG_USER
use_stderr = True
notification_driver =messaging
public_port=5000
admin_bind_host=192.168.0.24
public_bind_host=172.16.2.6
admin_port=35357
[assignment]
[auth]
[cache]
[catalog]
template_file = /etc/keystone/default_catalog.templates
driver = keystone.catalog.backends.sql.Catalog
[cors]
[cors.subdomain]
[credential]
[database]
connection = mysql://keystone:6wfzFyqWtvUksBaX8DZKX4ngt.2.4/keystone
idle_timeout = 3600
min_pool_size = 1
max_pool_size = 10
max_retries = 10
retry_interval = 10
max_overflow = 20
[domain_config]
[endpoint_filter]
[endpoint_policy]
[eventlet_server]
public_workers = 4
admin_workers = 4
[eventlet_server_ssl]
[federation]
[fernet_tokens]
key_repository = /etc/keystone/fernet-keys
[identity]
[identity_mapping]
[kvs]
[ldap]
[matchmaker_redis]
[matchmaker_ring]
[memcache]
[oauth1]
[os_inherit]
[oslo_messaging_amqp]
[oslo_messaging_qpid]
[oslo_messaging_rabbit]
rabbit_hosts = 172.16.2.6
rabbit_use_ssl = False
rabbit_userid = guest
rabbit_password = guest
rabbit_virtual_host = /
rabbit_ha_queues = True
heartbeat_timeout_threshold = 60
heartbeat_rate = 2
[oslo_middleware]
[oslo_policy]
[paste_deploy]
config_file = /usr/share/keystone/keystone-dist-paste.ini
[policy]
[resource]
[revoke]
driver = keystone.contrib.revoke.backends.sql.Revoke
[role]
[saml]
[signing]
certfile = /etc/keystone/ssl/certs/signing_cert.pem
keyfile = /etc/keystone/ssl/private/signing_key.pem
ca_certs = /etc/keystone/ssl/certs/ca.pem
ca_key = /etc/keystone/ssl/private/cakey.pem
key_size = 2048
cert_subject = /C=US/ST=Unset/L=Unset/O=Unset/CN=www.example.com
[ssl]
enable=False
[token]
expiration = 3600
provider = keystone.token.providers.uuid.Provider
driver = keystone.token.persistence.backends.sql.Token
revoke_by_id = True
[tokenless_auth]
[trust]
[ec2]
driver=keystone.contrib.ec2.backends.sql.Ec2

Controller nova.conf:

[DEFAULT]
novncproxy_host=172.16.2.6
novncproxy_port=6080
instance_name_template=instance-%08x
use_ipv6=False
notify_on_state_change=vm_and_task_state
notify_api_faults=False
state_path=/var/lib/nova
report_interval=10
enabled_apis=ec2,osapi_compute,metadata
ec2_listen=172.16.2.6
ec2_listen_port=8773
ec2_workers=4
osapi_compute_listen=172.16.2.6
osapi_compute_listen_port=8774
osapi_compute_workers=4
metadata_listen=172.16.2.6
metadata_listen_port=8775
metadata_workers=4
service_down_time=60
rootwrap_config=/etc/nova/rootwrap.conf
volume_api_class=nova.volume.cinder.API
api_paste_config=api-paste.ini
auth_strategy=keystone
use_forwarded_for=False
fping_path=/usr/sbin/fping
cpu_allocation_ratio=16.0
ram_allocation_ratio=1.0
network_api_class=nova.network.neutronv2.api.API
default_floating_pool=public
dhcp_domain=
security_group_api=neutron
memcached_servers=172.16.2.6:11211
disk_allocation_ratio=1.0
max_io_ops_per_host=8
max_instances_per_host=50
scheduler_host_manager=nova.scheduler.host_manager.HostManager
scheduler_host_subset_size=1
scheduler_available_filters=nova.scheduler.filters.all_filters
scheduler_weight_classes=nova.scheduler.weights.all_weighers
scheduler_use_baremetal_filters=False
scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
scheduler_max_attempts=3
vif_plugging_is_fatal=True
vif_plugging_timeout=300
firewall_driver=nova.virt.firewall.NoopFirewallDriver
debug=False
verbose=False
log_dir=/var/log/nova
use_syslog=False
syslog_log_facility=LOG_USER
use_stderr=True
notification_topics=notifications
rpc_backend=rabbit
image_service=nova.image.glance.GlanceImageService
lock_path=/var/lib/nova/tmp
osapi_volume_listen=172.16.2.6
novncproxy_base_url=http://172.16.2.6:6080/vnc_auto.html
[api_database]
[barbican]
[cells]
[cinder]
catalog_info=volumev2:cinderv2:internalURL
[conductor]
use_local=False
[cors]
[cors.subdomain]
[database]
connection=mysql://nova:Vt7PPdyMVtCsNFUmzb9ztZ3cV.2.4/nova
idle_timeout=3600
min_pool_size=1
max_pool_size=10
max_retries=10
retry_interval=10
max_overflow=20
[ephemeral_storage_encryption]
[glance]
api_servers=http://172.16.1.4:9292
[guestfs]
[hyperv]
[image_file_url]
[ironic]
[keymgr]
[keystone_authtoken]
auth_uri=http://172.16.2.4:5000/v2.0
identity_uri=http://192.168.0.22:35357
admin_user=nova
admin_password=Vt7PPdyMVtCsNFUmzb9ztZ3cV
admin_tenant_name=service
[libvirt]
[matchmaker_redis]
[matchmaker_ring]
[metrics]
[neutron]
service_metadata_proxy=True
metadata_proxy_shared_secret =GG8PhzrZfGrgNHZ8RGHCm83AP
url=http://172.16.2.4:9696
admin_username=neutron
admin_password=pwwaAtn44xdewfTtgfRkVQ4EX
admin_tenant_name=service
region_name=RegionOne
admin_auth_url=http://192.168.0.22:35357
auth_strategy=keystone
ovs_bridge=br-int
extension_sync_interval=600
timeout=30
default_tenant_id=default
[osapi_v21]
[oslo_concurrency]
[oslo_messaging_amqp]
[oslo_messaging_qpid]
[oslo_messaging_rabbit]
amqp_durable_queues=False
kombu_reconnect_delay=1.0
rabbit_hosts=172.16.2.6
rabbit_use_ssl=False
rabbit_userid=guest
rabbit_password=guest
rabbit_virtual_host=/
rabbit_ha_queues=True
heartbeat_timeout_threshold=60
heartbeat_rate=2
[oslo_middleware]
[rdp]
[serial_console]
[spice]
[ssl]
[trusted_computing]
[upgrade_levels]
[vmware]
[vnc]
[workarounds]
[xenserver]
[zookeeper]
[osapi_v3]
enabled=True

Compute nova.conf

[DEFAULT]
internal_service_availability_zone=internal
default_availability_zone=nova
my_ip=172.16.2.7
use_ipv6=False
notify_on_state_change=vm_and_task_state
notify_api_faults=False
state_path=/var/lib/nova
report_interval=10
compute_manager=nova.compute.manager.ComputeManager
service_down_time=60
instance_usage_audit_period=hour
rootwrap_config=/etc/nova/rootwrap.conf
auth_strategy=keystone
allow_resize_to_same_host=False
instance_usage_audit=True
heal_instance_info_cache_interval=60
reserved_host_memory_mb=512
network_api_class=nova.network.neutronv2.api.API
linuxnet_interface_driver=nova.network.linux_net.LinuxOVSInterfaceDriver
dhcp_domain=
security_group_api=neutron
compute_driver=libvirt.LibvirtDriver
vif_plugging_is_fatal=True
vif_plugging_timeout=300
firewall_driver=nova.virt.firewall.NoopFirewallDriver
force_raw_images=True
debug=False
verbose=False
log_dir=/var/log/nova
use_syslog=False
syslog_log_facility=LOG_USER
use_stderr=True
notification_topics=notifications
rpc_backend=rabbit
vncserver_proxyclient_address=172.16.2.7
vnc_keymap=en-us
vnc_enabled=True
image_service=nova.image.glance.GlanceImageService
lock_path=/var/lib/nova/tmp
vncserver_listen=0.0.0.0
novncproxy_base_url=http://10.8.68.111:6080/vnc_auto.html
[api_database]
[barbican]
[cells]
[cinder]
catalog_info=volumev2:cinderv2:internalURL
[conductor]
[cors]
[cors.subdomain]
[database]
[ephemeral_storage_encryption]
[glance]
api_servers=http://172.16.1.4:9292
[guestfs]
[hyperv]
[image_file_url]
[ironic]
[keymgr]
[keystone_authtoken]
[libvirt]
virt_type=kvm
inject_password=False
inject_key=False
inject_partition=-2
cpu_mode=host-model
[matchmaker_redis]
[matchmaker_ring]
[metrics]
[neutron]
url=http://172.16.2.4:9696
admin_username=neutron
admin_password=pwwaAtn44xdewfTtgfRkVQ4EX
admin_tenant_name=service
region_name=RegionOne
admin_auth_url=http://192.168.0.22:35357/v2.0
auth_strategy=keystone
ovs_bridge=br-int
extension_sync_interval=600
timeout=30
default_tenant_id=default
[osapi_v21]
[oslo_concurrency]
[oslo_messaging_amqp]
[oslo_messaging_qpid]
[oslo_messaging_rabbit]
amqp_durable_queues=False
kombu_reconnect_delay=1.0
rabbit_hosts=172.16.2.6
rabbit_use_ssl=False
rabbit_userid=guest
rabbit_password=guest
rabbit_virtual_host=/
rabbit_ha_queues=True
heartbeat_timeout_threshold=60
heartbeat_rate=2
[oslo_middleware]
[rdp]
[serial_console]
[spice]
[ssl]
[trusted_computing]
[upgrade_levels]
[vmware]
[vnc]
[workarounds]
[xenserver]
[zookeeper]

The logs from nova and keystone, compute and controller is attached

Created attachment 1132877 [details]
keystone log from controller

Created attachment 1132878 [details]
nova-api log from controller

Created attachment 1132879 [details]
nova-compute from compute node

A couple more things:

  - Can you reproduce this issue _without_ BigSwitch plugin?  That will 
    allow us narrow down the issue (I don't have the hardware to try
    to reproduce this myself)

  - Also, you might want to provide the Neutron network configuration  
    details (including BigSwitch plugin specific configs) -- so 
    Networking experts might be able to look at.

Hi
I tried with vanilla installation, with linux bonding configured and still seeing the same issue

below is the log and neutron.conf and ml2_conf.ini file. Let me know if something else is needed


2016-03-04 17:04:37.026 2943 WARNING oslo_config.cfg [-] Option "novncproxy_base_url" from group "DEFAULT" is deprecated. Use option "novncproxy_base_url" from group "vnc".
2016-03-04 17:32:05.980 3541 WARNING oslo_config.cfg [req-b7d16a6a-9b18-4b08-99f9-0555a818469e 284a376ac56e4aa1a025a87d9fb8d4b4 1c0974c84f3f410d9dbac843db95b226 - - -] Option "auth_strategy" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
2016-03-04 17:32:05.981 3541 WARNING oslo_config.cfg [req-b7d16a6a-9b18-4b08-99f9-0555a818469e 284a376ac56e4aa1a025a87d9fb8d4b4 1c0974c84f3f410d9dbac843db95b226 - - -] Option "admin_auth_url" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
2016-03-04 17:32:05.981 3541 WARNING oslo_config.cfg [req-b7d16a6a-9b18-4b08-99f9-0555a818469e 284a376ac56e4aa1a025a87d9fb8d4b4 1c0974c84f3f410d9dbac843db95b226 - - -] Option "admin_username" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
2016-03-04 17:32:05.982 3541 WARNING oslo_config.cfg [req-b7d16a6a-9b18-4b08-99f9-0555a818469e 284a376ac56e4aa1a025a87d9fb8d4b4 1c0974c84f3f410d9dbac843db95b226 - - -] Option "admin_password" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
2016-03-04 17:32:05.982 3541 WARNING oslo_config.cfg [req-b7d16a6a-9b18-4b08-99f9-0555a818469e 284a376ac56e4aa1a025a87d9fb8d4b4 1c0974c84f3f410d9dbac843db95b226 - - -] Option "admin_tenant_name" from group "neutron" is deprecated for removal.  Its value may be silently ignored in the future.
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions [req-b7d16a6a-9b18-4b08-99f9-0555a818469e 284a376ac56e4aa1a025a87d9fb8d4b4 1c0974c84f3f410d9dbac843db95b226 - - -] Unexpected exception in API method
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 611, in create
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     **create_kwargs)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 149, in inner
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1581, in create
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1181, in _create_instance
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     auto_disk_config, reservation_id, max_count)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 955, in _validate_and_build_base_options
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     pci_request_info, requested_networks)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1059, in create_pci_requests_for_sriov_ports
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     neutron = get_client(context, admin=True)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 237, in get_client
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     auth_token = _ADMIN_AUTH.get_token(_SESSION)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     return self.get_access(session).auth_token
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     self.auth_ref = self.get_auth_ref(session)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py", line 88, in get_auth_ref
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     authenticated=False, log=False)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     return self.request(url, 'POST', **kwargs)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions     raise exceptions.from_response(resp, method, url)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions NotFound: The resource could not be found. (HTTP 404)
2016-03-04 17:32:05.986 3541 ERROR nova.api.openstack.extensions


neutron.conf
[DEFAULT]
verbose = False
router_distributed = False
debug = False
state_path = /var/lib/neutron
use_syslog = False
use_stderr = True
log_dir =/var/log/neutron
bind_host = 172.16.2.7
bind_port = 9696
core_plugin =ml2
service_plugins =router
auth_strategy = keystone
base_mac = fa:16:3e:00:00:00
mac_generation_retries = 16
dhcp_lease_duration = 86400
dhcp_agent_notification = True
allow_bulk = True
allow_pagination = False
allow_sorting = False
allow_overlapping_ips = True
advertise_mtu = False
agent_down_time = 75
router_scheduler_driver = neutron.scheduler.l3_agent_scheduler.ChanceScheduler
allow_automatic_l3agent_failover = False
dhcp_agents_per_network = 1
l3_ha = False
api_workers = 4
rpc_workers = 4
use_ssl = False
notify_nova_on_port_status_changes = True
notify_nova_on_port_data_changes = True
nova_url = http://172.16.2.5:8774/v2/
send_events_interval = 2
rpc_response_timeout=60
rpc_backend=rabbit
control_exchange=neutron
lock_path=/var/lib/neutron/lock
[matchmaker_redis]
[matchmaker_ring]
[quotas]
[agent]
root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf
report_interval = 30
[keystone_authtoken]
auth_uri = http://172.16.2.5:5000/v2.0
identity_uri = http://192.168.0.6:35357
admin_tenant_name = service
admin_user = neutron
admin_password = hZYQNjHTZj8ZNu4BT9WVMXhKr
[database]
connection = mysql://neutron:hZYQNjHTZj8ZNu4BT9WVMXhKr.2.5/ovs_neutron?charset=utf8
max_retries = 10
retry_interval = 10
min_pool_size = 1
max_pool_size = 10
idle_timeout = 3600
max_overflow = 20
[nova]
auth_plugin =password
project_name=service
username=nova
password=Pn8RDsP4nFTGWdWCXwRZwaJrY
project_domain_id=default
tenant_name=service
user_domain_id=default
auth_url=http://192.168.0.6:35357
[oslo_concurrency]
[oslo_policy]
[oslo_messaging_amqp]
[oslo_messaging_qpid]
[oslo_messaging_rabbit]
kombu_reconnect_delay = 1.0
rabbit_hosts = 172.16.2.7
rabbit_use_ssl = False
rabbit_userid = guest
rabbit_password = guest
rabbit_virtual_host = /
rabbit_ha_queues = True
heartbeat_rate=2
heartbeat_timeout_threshold=60
[qos]
notification_drivers = message_queue

ml2_conf.ini file
[ml2]
type_drivers = vxlan,vlan,flat,gre
tenant_network_types = vlan
mechanism_drivers =openvswitch
extension_drivers =
path_mtu = 0
[ml2_type_flat]
flat_networks =datacentre
[ml2_type_vlan]
network_vlan_ranges =datacentre:1001:2000
[ml2_type_gre]
tunnel_id_ranges =1:1000
[ml2_type_vxlan]
vni_ranges =1:1000
vxlan_group =224.0.0.1
[ml2_type_geneve]
[securitygroup]
enable_security_group = True

Hi,

I'm running into the same problem with OSP 8 Beta 7 on 3x Controller + 3x Compute Node configuration *without* any extra plugins, using vxlan configuration.

Here is the command I have uesed to create the oovercloud:

openstack overcloud deploy --stack osp8r1 --templates ~/templates/my-overcloud -e ~/templates/my-overcloud/environments/network-isolation.yaml -e ~/templates/network-environment.yaml -e ~/templates/my-overcloud/rhel-registration/environment-rhel-registration.yaml -e ~/templates/my-overcloud/rhel-registration/rhel-registration-resource-registry.yaml  -e ~/templates/storage-environment.yaml  -e  ~/templates/pre-deployment.yaml -e ~/templates/post-deployment.yaml --control-scale 3 --compute-scale 3 --control-flavor control --compute-flavor compute --ntp-server time.pdb.fsc.net --neutron-network-type vxlan --neutron-tunnel-types vxlan

When I try to create an instance I saw the following error:

[stack@osp8dr1 ~(OC-admin)]$ nova boot --flavor fjsv.rhel_xl --boot-volume 934ef6b8-4bd8-4177-b262-9acc3143644f --key-name key-admin --nic net-id=d1db1bfd-005c-408b-bad5-9dcd32599d24 --security-group sec1 --availability-zone nova:osp8r1-novacompute-0.localdomain vmA1_1
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'keystoneclient.exceptions.NotFound'> (HTTP 500) (Request-ID: req-d5939278-7245-435b-9e16-3b425a1357f7)

In the API log from controller-1 I have the following traceback:

2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions [req-d5939278-7245-435b-9e16-3b425a1357f7 0d96230dfeeb46cd858141e0459ffee9 5e7f78267faa4832a292d65f220bca19 - - -] Unexpected exception in API method
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 611, in create
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     **create_kwargs)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 149, in inner
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1581, in create
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1181, in _create_instance
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     auto_disk_config, reservation_id, max_count)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 955, in _validate_and_build_base_options
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     pci_request_info, requested_networks)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1059, in create_pci_requests_for_sriov_ports
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     neutron = get_client(context, admin=True)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 237, in get_client
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     auth_token = _ADMIN_AUTH.get_token(_SESSION)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     return self.get_access(session).auth_token
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     self.auth_ref = self.get_auth_ref(session)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py", line 88, in get_auth_ref
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     authenticated=False, log=False)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     return self.request(url, 'POST', **kwargs)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions     raise exceptions.from_response(resp, method, url)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions NotFound: The resource could not be found. (HTTP 404)
2016-03-08 12:08:37.286 5065 ERROR nova.api.openstack.extensions

I will attach sosreport from all 3 Controller-nodes.

Regards, 

Ralf (Fujitsu)

Created attachment 1134103 [details]
sosreport from controller-1

Hi,
I think this ticket is actually https://bugzilla.redhat.com/show_bug.cgi?id=1308422
There is a workaround until the fix will get in
see comment #4
Workaround: 
change the admin_auth_url to  http://192.0.2.18:35357/v2.0 on all controllers.
pcs resource restart openstack-nova-api-clone

Hello,

Looking at both of the provided nova.conf files, it appears that you are using a wrong keystone endpoint. (Missing /v2.0 at the end of your admin_auth_url)

Please follow comment 11 for the workaround.

Thanks,
Vladik

Hi,
I am able to launch instance after following comment 11. Please close this ticket.


Thanks

Song

Closing this bug as per comments #11, #12, #13.  (Thanks Nathan & Vladik for confirming).  

Just re-stating the root cause:

Explicitly specifying the Keystone end point version as part of 'admin_auth_url' in nova.conf resolves this:

    admin_auth_url=http://$IP-ADDRESS:35357/v2.0