Bug 1388521

Summary: auth_plugin configuration is not specified after OSP8->OSP9 upgrade
Product: Red Hat OpenStack Reporter: kamil.rogon
Component: openstack-tripleo-heat-templatesAssignee: Sofer Athlan-Guyot <sathlang>
Status: CLOSED DUPLICATE QA Contact: Arik Chernetsky <achernet>
Severity: unspecified Docs Contact:
Priority: urgent    
Version: 9.0 (Mitaka)CC: djuran, jcoufal, jdonohue, jslagle, mburns, mcornea, rhel-osp-director-maint, sathlang
Target Milestone: ---Keywords: ZStream
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-17 10:39:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1373538, 1400606    

Description kamil.rogon 2016-10-25 14:45:45 UTC 
Description of problem:
During one of the upgrade steps I have included additional template called "major-upgrade-pacemaker.yaml". After this step I am not able to boot new instance on overcloud. It seems like the auth_plugin parameter (under [neutron] section in nova.conf) is not set. Nova-api throws the following traceback:

2016-10-25 13:58:11.204 20299 WARNING keystonemiddleware.auth_token [-] Using the in-process token cache is deprecated as of the 4.2.0 release and may be removed in the 5.0.0 release or the 'O' development cycle. The in-process cache causes inconsistent results and high memory usage. When the feature is removed the auth_token middleware will not cache tokens by default which may result in performance issues. It is recommended to use  memcache for the auth_token token cache by setting the memcached_servers option.
2016-10-25 13:58:11.206 20299 WARNING oslo_config.cfg [-] Option "memcached_servers" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions [req-d17a7110-2fb3-473f-8dda-0540d8176316 3de032dcc73c4c418fb02c5634d279c3 da845199f7fd4532adc6904aaebe3350 - - -] Unexpected exception in API method
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 629, in create
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions     **create_kwargs)
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1563, in create
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1146, in _create_instance
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions     reservation_id, max_count)
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 873, in _validate_and_build_base_options
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions     pci_request_info, requested_networks)
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1125, in create_pci_requests_for_sriov_ports
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions     neutron = get_client(context, admin=True)
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 183, in get_client
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions     _ADMIN_AUTH = _load_auth_plugin(CONF)
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 122, in _load_auth_plugin
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions     raise neutron_client_exc.Unauthorized(message=err_msg)
2016-10-25 13:58:14.062 20327 ERROR nova.api.openstack.extensions Unauthorized: Unknown auth type: None


Version-Release number of selected component (if applicable):
[root@overcloud-controller-0 ~]# rpm -qa | grep nova
openstack-nova-scheduler-13.1.1-7.el7ost.noarch
openstack-nova-compute-13.1.1-7.el7ost.noarch
openstack-nova-cert-13.1.1-7.el7ost.noarch
openstack-nova-common-13.1.1-7.el7ost.noarch
openstack-nova-console-13.1.1-7.el7ost.noarch
openstack-nova-conductor-13.1.1-7.el7ost.noarch
python-novaclient-3.3.2-1.el7ost.noarch
openstack-nova-api-13.1.1-7.el7ost.noarch
python-nova-13.1.1-7.el7ost.noarch
openstack-nova-novncproxy-13.1.1-7.el7ost.noarch

How reproducible:


Steps to Reproduce:
1. execute major version upgrade from director including "major-upgrade-pacemaker.yaml" step.
2. Try to boot VM from director using
[stack@osp9-director ~]$ source overcloudrc
[stack@osp8-director ~]$ nova --debug boot --flavor 2 --image rhel7 --nic net-id=cc5c67be-1ce0-4606-9d63-5f31fe361ffa after_step5

Actual results:
DEBUG (session:248) REQ: curl -g -i -X POST http://10.2.116.50:8774/v2.1/da845199f7fd4532adc6904aaebe3350/servers -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.25" -H "X-Auth-Token: {SHA1}afb1d5fe6c9f31f4d5921fd2b102aa4360f1b4ee" -d '{"server": {"name": "after_step5", "imageRef": "7cba2c2f-a87c-420d-b05b-fcd790c0d11b", "flavorRef": "2", "max_count": 1, "min_count": 1, "networks": [{"uuid": "cc5c67be-1ce0-4606-9d63-5f31fe361ffa"}]}}'
DEBUG (connectionpool:395) "POST /v2.1/da845199f7fd4532adc6904aaebe3350/servers HTTP/1.1" 500 218
DEBUG (session:277) RESP: [500] X-Openstack-Nova-Api-Version: 2.25 Vary: X-OpenStack-Nova-API-Version Content-Type: application/json; charset=UTF-8 Content-Length: 218 X-Compute-Request-Id: req-d17a7110-2fb3-473f-8dda-0540d8176316 Date: Tue, 25 Oct 2016 13:58:14 GMT
RESP BODY: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'neutronclient.common.exceptions.Unauthorized'>", "code": 500}}

DEBUG (shell:1082) Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'neutronclient.common.exceptions.Unauthorized'> (HTTP 500) (Request-ID: req-d17a7110-2fb3-473f-8dda-0540d8176316)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 1080, in main
    OpenStackComputeShell().main(argv)
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 1007, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/shell.py", line 589, in do_boot
    server = cs.servers.create(*boot_args, **boot_kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1233, in create
    **boot_kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 667, in _boot
    return_raw=return_raw, **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 345, in _create
    resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 179, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 94, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'neutronclient.common.exceptions.Unauthorized'> (HTTP 500) (Request-ID: req-d17a7110-2fb3-473f-8dda-0540d8176316)


Expected results:
VM started (available as building/active in "nova list").


Additional info:
Comment 1 kamil.rogon 2016-11-18 07:04:59 UTC 
The missing configuration on controllers seems to be:
# openstack-config --set /etc/nova/nova.conf neutron auth_plugin v3password
# openstack-config --set /etc/nova/nova.conf neutron auth_url http://$ADMIN_API_VIP:35357/v3
Comment 5 Sofer Athlan-Guyot 2017-01-25 00:50:51 UTC 
Hi,

this is related to https://bugzilla.redhat.com/show_bug.cgi?id=1385143

The parameter you've found are corrects, the full list of parameters to update are shown there https://review.openstack.org/#/c/422837/4/extraconfig/tasks/liberty_to_mitaka_neutron_auth_param.sh

We have a patch https://bugzilla.redhat.com/show_bug.cgi?id=1385143#c82 .  And should be in version openstack-tripleo-heat-templates-2.0.0-43.el7ost

But this patch will make compute work just after the you upgrade a compute with the non-controller script, so it may not be enough for the problem at hand.  Can you confirm that the compute node were not updated ?

Regards,
Comment 6 kamil.rogon 2017-01-25 08:36:17 UTC 
Hi,
I do not have permission to both Bugzilla bugs, that you have mentioned in your comment. However glad that it is going to be fixed in upcoming version.

Regarding your question, the computes were not updated and you are right that the settings have to be applied on them also. For compute configuration I have created related bug: https://bugzilla.redhat.com/show_bug.cgi?id=1396365
Comment 7 Sofer Athlan-Guyot 2017-01-26 11:02:41 UTC 
Hi,

sorry about the permissions problem, but the main points are that we have a fix coming (but it needs improvement to support you use case)  and the parameters are those in the openstack review.

Will keep you informed of the upcoming patch.

Regards,
Comment 10 Sofer Athlan-Guyot 2017-03-17 10:39:28 UTC 
Hi,

fixed in openstack-tripleo-heat-templates-2.0.0-45.el7ost

Closing this bug as a duplicate.  If you have any trouble to get the relevant package don't hesitate to re-open the bug.

Regards,

*** This bug has been marked as a duplicate of bug 1413686 ***