Bug 1396365

Summary: Unauthorized: Unknown auth type: None: unable to boot instance after OSP8->OSP9 upgrade
Product: Red Hat OpenStack Reporter: kamil.rogon
Component: openstack-tripleo-heat-templatesAssignee: Sofer Athlan-Guyot <sathlang>
Status: CLOSED DUPLICATE QA Contact: Arik Chernetsky <achernet>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.0 (Mitaka)CC: aschultz, djuran, mburns, mcornea, rhel-osp-director-maint, sathlang
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-17 10:18:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description kamil.rogon 2016-11-18 07:54:34 UTC 
Description of problem:
See similar bug for controller misconfiguration: https://bugzilla.redhat.com/show_bug.cgi?id=1388521


Version-Release number of selected component (if applicable):
[root@osp9-director ~]# rpm -qa | grep heat
openstack-heat-templates-0-0.3.96a0b0bgit.el7ost.noarch
openstack-tripleo-heat-templates-kilo-0.8.14-16.el7ost.noarch
openstack-heat-engine-6.0.0-11.el7ost.noarch
openstack-tripleo-heat-templates-liberty-2.0.0-34.el7ost.noarch
openstack-heat-api-cfn-6.0.0-11.el7ost.noarch
openstack-heat-api-cloudwatch-6.0.0-11.el7ost.noarch
openstack-heat-common-6.0.0-11.el7ost.noarch
openstack-tripleo-heat-templates-2.0.0-34.el7ost.noarch
python-heatclient-1.2.0-1.el7ost.noarch
openstack-heat-api-6.0.0-11.el7ost.noarch
python-heat-tests-6.0.0-11.el7ost.noarch

[root@overcloud-compute-4 ~]# rpm -qa | grep nova
python-novaclient-3.3.2-1.el7ost.noarch
openstack-nova-console-13.1.1-7.el7ost.noarch
openstack-nova-novncproxy-13.1.1-7.el7ost.noarch
python-nova-13.1.1-7.el7ost.noarch
openstack-nova-scheduler-13.1.1-7.el7ost.noarch
openstack-nova-api-13.1.1-7.el7ost.noarch
openstack-nova-compute-13.1.1-7.el7ost.noarch
openstack-nova-conductor-13.1.1-7.el7ost.noarch
openstack-nova-common-13.1.1-7.el7ost.noarch
openstack-nova-cert-13.1.1-7.el7ost.noarch

How reproducible:


Steps to Reproduce:
1. execute major version upgrade from director including "major-upgrade-pacemaker.yaml" step.
2. Try to boot VM from director using
[stack@osp9-director ~]$ source overcloudrc
[stack@osp8-director ~]$ nova --debug boot --flavor 2 --image rhel7 --nic net-id=cc5c67be-1ce0-4606-9d63-5f31fe361ffa novacompute_test

Actual results:
VM is scheduled but fails to build. Traceback in nova-compute.log with message: Unauthorized: Unknown auth type: None

Expected results:
VM properly started


Additional info:
Traceback from nova-compute.log:
2016-11-18 07:44:19.880 19586 ERROR nova.compute.resource_tracker [req-02d8f0c9-7082-4dc6-a567-0caefa489ec0 - - - - -] Migration for instance 9bc012b9-ac25-41a8-9aa6-d75934f72ff2 refers to a
nother host's instance!
2016-11-18 07:44:19.924 19586 INFO nova.compute.resource_tracker [req-02d8f0c9-7082-4dc6-a567-0caefa489ec0 - - - - -] Total usable vcpus: 40, total allocated vcpus: 2
2016-11-18 07:44:19.924 19586 INFO nova.compute.resource_tracker [req-02d8f0c9-7082-4dc6-a567-0caefa489ec0 - - - - -] Final resource view: name=overcloud-compute-3.localdomain phys_ram=26187
8MB used_ram=6144MB phys_disk=131952GB used_disk=40GB total_vcpus=40 used_vcpus=2 pci_stats=[]
2016-11-18 07:44:19.962 19586 INFO nova.compute.resource_tracker [req-02d8f0c9-7082-4dc6-a567-0caefa489ec0 - - - - -] Compute_service record updated for overcloud-compute-3.localdomain:overc
loud-compute-3.localdomain
2016-11-18 07:44:20.050 19586 ERROR nova.compute.manager [req-02d8f0c9-7082-4dc6-a567-0caefa489ec0 - - - - -] [instance: fbebea65-f8a2-43fe-aa1f-86d38940e1c9] An error occurred while refresh
ing the network cache.
2016-11-18 07:44:20.050 19586 ERROR nova.compute.manager [instance: fbebea65-f8a2-43fe-aa1f-86d38940e1c9] Traceback (most recent call last):
2016-11-18 07:44:20.050 19586 ERROR nova.compute.manager [instance: fbebea65-f8a2-43fe-aa1f-86d38940e1c9]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5808, in _he
al_instance_info_cache
2016-11-18 07:44:20.050 19586 ERROR nova.compute.manager [instance: fbebea65-f8a2-43fe-aa1f-86d38940e1c9]     self.network_api.get_instance_nw_info(context, instance)
2016-11-18 07:44:20.050 19586 ERROR nova.compute.manager [instance: fbebea65-f8a2-43fe-aa1f-86d38940e1c9]   File "/usr/lib/python2.7/site-packages/nova/network/base_api.py", line 253, in get
_instance_nw_info
2016-11-18 07:44:20.050 19586 ERROR nova.compute.manager [instance: fbebea65-f8a2-43fe-aa1f-86d38940e1c9]     result = self._get_instance_nw_info(context, instance, **kwargs)
2016-11-18 07:44:20.050 19586 ERROR nova.compute.manager [instance: fbebea65-f8a2-43fe-aa1f-86d38940e1c9]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 978, i
n _get_instance_nw_info
2016-11-18 07:44:20.050 19586 ERROR nova.compute.manager [instance: fbebea65-f8a2-43fe-aa1f-86d38940e1c9]     preexisting_port_ids)
2016-11-18 07:44:20.050 19586 ERROR nova.compute.manager [instance: fbebea65-f8a2-43fe-aa1f-86d38940e1c9]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1749,
in _build_network_info_model
2016-11-18 07:44:20.050 19586 ERROR nova.compute.manager [instance: fbebea65-f8a2-43fe-aa1f-86d38940e1c9]     client = get_client(context, admin=True)
2016-11-18 07:44:20.050 19586 ERROR nova.compute.manager [instance: fbebea65-f8a2-43fe-aa1f-86d38940e1c9]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 183, i
n get_client
2016-11-18 07:44:20.050 19586 ERROR nova.compute.manager [instance: fbebea65-f8a2-43fe-aa1f-86d38940e1c9]     _ADMIN_AUTH = _load_auth_plugin(CONF)
2016-11-18 07:44:20.050 19586 ERROR nova.compute.manager [instance: fbebea65-f8a2-43fe-aa1f-86d38940e1c9]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 122, i
n _load_auth_plugin
2016-11-18 07:44:20.050 19586 ERROR nova.compute.manager [instance: fbebea65-f8a2-43fe-aa1f-86d38940e1c9]     raise neutron_client_exc.Unauthorized(message=err_msg)
2016-11-18 07:44:20.050 19586 ERROR nova.compute.manager [instance: fbebea65-f8a2-43fe-aa1f-86d38940e1c9] Unauthorized: Unknown auth type: None
2016-11-18 07:44:20.050 19586 ERROR nova.compute.manager [instance: fbebea65-f8a2-43fe-aa1f-86d38940e1c9]
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [req-8e0e40ba-2754-4dd9-8c6e-d70d465cb0a8 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: 0dee658
2-01b0-4770-8a41-f762689553e2] Instance failed to spawn
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2] Traceback (most recent call last):
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2218, in _bu
ild_resources
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]     yield resources
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2064, in _bu
ild_and_run_instance
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]     block_device_info=block_device_info)
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2780, in
 spawn
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]     write_to_disk=True)
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4719, in
 _get_guest_xml
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]     network_info_str = str(network_info)
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 521, in __str_
_
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]     return self._sync_wrapper(fn, *args, **kwargs)
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 504, in _sync_
wrapper
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]     self.wait()
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]   File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 536, in wait
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]     self[:] = self._gt.wait()
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]     return self._exit_event.wait()
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]     current.throw(*self._exc)
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]     result = function(*args, **kwargs)
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 1145, in context_wrapp
er
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]     return func(*args, **kwargs)
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1587, in _al
locate_network_async
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]     six.reraise(*exc_info)
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1570, in _al
locate_network_async
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]     bind_host_id=bind_host_id)
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 611, i
n allocate_for_instance
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]     get_client(context, admin=True))
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 183, i
n get_client
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]     _ADMIN_AUTH = _load_auth_plugin(CONF)
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 122, i
n _load_auth_plugin
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]     raise neutron_client_exc.Unauthorized(message=err_msg)
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2] Unauthorized: Unknown auth type: None
2016-11-18 07:44:28.767 19586 ERROR nova.compute.manager [instance: 0dee6582-01b0-4770-8a41-f762689553e2]
2016-11-18 07:44:28.770 19586 INFO nova.compute.manager [req-8e0e40ba-2754-4dd9-8c6e-d70d465cb0a8 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: 0dee6582
-01b0-4770-8a41-f762689553e2] Terminating instance
2016-11-18 07:44:28.776 19586 INFO nova.virt.libvirt.driver [-] [instance: 0dee6582-01b0-4770-8a41-f762689553e2] During wait destroy, instance disappeared.
2016-11-18 07:44:29.097 19586 INFO nova.virt.libvirt.driver [req-8e0e40ba-2754-4dd9-8c6e-d70d465cb0a8 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: 0dee
6582-01b0-4770-8a41-f762689553e2] Deleting instance files /var/lib/nova/instances/0dee6582-01b0-4770-8a41-f762689553e2_del
2016-11-18 07:44:29.097 19586 INFO nova.virt.libvirt.driver [req-8e0e40ba-2754-4dd9-8c6e-d70d465cb0a8 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: 0dee
6582-01b0-4770-8a41-f762689553e2] Deletion of /var/lib/nova/instances/0dee6582-01b0-4770-8a41-f762689553e2_del complete
2016-11-18 07:44:29.361 19586 INFO nova.compute.manager [req-8e0e40ba-2754-4dd9-8c6e-d70d465cb0a8 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: 0dee6582
-01b0-4770-8a41-f762689553e2] Took 0.59 seconds to destroy the instance on the hypervisor.
2016-11-18 07:44:29.397 19586 ERROR nova.compute.manager [req-8e0e40ba-2754-4dd9-8c6e-d70d465cb0a8 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] [instance: 0dee658
2-01b0-4770-8a41-f762689553e2] Failed to deallocate network for instance.
Comment 1 kamil.rogon 2016-11-18 07:55:24 UTC 
The missing configuration on computes seems to be:
# openstack-config --set /etc/nova/nova.conf neutron auth_plugin v3password
# openstack-config --set /etc/nova/nova.conf neutron auth_url http://$ADMIN_API_VIP:35357/v3

Restart nova-compute by:
# systemctl restart openstack-nova-compute
Comment 2 Sofer Athlan-Guyot 2017-03-17 10:18:45 UTC 
Hi,

fixed in openstack-tripleo-heat-templates-2.0.0-45.el7ost

Closing this bug as a duplicate.  If you have any trouble to get the relevant package don't hesitate to re-open the bug.

Regards,

*** This bug has been marked as a duplicate of bug 1413686 ***