Bug 1291106

Description of problem:

Instance HA relies in part on the nova.services.force_down() command.
This is not currently available on OSP8.

Version-Release number of selected component (if applicable):

openstack-nova-api-12.0.0-2.el7ost.noarch
python-nova-12.0.0-2.el7ost.noarch
openstack-nova-common-12.0.0-2.el7ost.noarch
openstack-nova-compute-12.0.0-2.el7ost.noarch
openstack-nova-scheduler-12.0.0-2.el7ost.noarch
openstack-nova-conductor-12.0.0-2.el7ost.noarch
openstack-nova-cert-12.0.0-2.el7ost.noarch
openstack-nova-novncproxy-12.0.0-2.el7ost.noarch
python-novaclient-2.30.1-1.el7ost.noarch
openstack-nova-console-12.0.0-2.el7ost.noarch

How reproducible:

100%

Steps to Reproduce:

export OS_NO_CACHE=True
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export no_proxy=,172.20.0.10,172.20.0.10,172.20.0.10
export OS_TENANT_NAME=admin
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=http://172.20.0.10:5000/v2.1/
export NOVA_VERSION=2.11
export OS_PASSWORD=...

1. nova service-force-down ${HOST} nova-compute
2.
3.

Actual results:

[root@overcloud-controller-0 heat-admin]# nova service-force-down overcloud-novacompute-1.localdomain nova-compute
No handlers could be found for logger "keystoneclient.auth.identity.generic.base"
ERROR (DiscoveryFailure): Could not determine a suitable URL for the plugin

or, with OS_AUTH_URL=http://172.20.0.10:5000/v2.0/

[root@overcloud-controller-0 heat-admin]# nova service-force-down overcloud-novacompute-1.localdomain nova-compute
usage: nova [--version] [--debug] [--os-cache] [--timings]
            [--os-auth-token OS_AUTH_TOKEN]
            [--os-tenant-name <auth-tenant-name>]
            [--os-tenant-id <auth-tenant-id>] [--os-region-name <region-name>]
            [--os-auth-system <auth-system>] [--service-type <service-type>]
            [--service-name <service-name>]
            [--volume-service-name <volume-service-name>]
            [--os-endpoint-type <endpoint-type>]
            [--os-compute-api-version <compute-api-ver>]
            [--bypass-url <bypass-url>] [--insecure]
            [--os-cacert <ca-certificate>] [--os-cert <certificate>]
            [--os-key <key>] [--timeout <seconds>] [--os-auth-url OS_AUTH_URL]
            [--os-domain-id OS_DOMAIN_ID] [--os-domain-name OS_DOMAIN_NAME]
            [--os-project-id OS_PROJECT_ID]
            [--os-project-name OS_PROJECT_NAME]
            [--os-project-domain-id OS_PROJECT_DOMAIN_ID]
            [--os-project-domain-name OS_PROJECT_DOMAIN_NAME]
            [--os-trust-id OS_TRUST_ID] [--os-user-id OS_USER_ID]
            [--os-user-name OS_USERNAME]
            [--os-user-domain-id OS_USER_DOMAIN_ID]
            [--os-user-domain-name OS_USER_DOMAIN_NAME]
            [--os-password OS_PASSWORD]
            <subcommand> ...
error: argument <subcommand>: invalid choice: u'service-force-down'
Try 'nova help ' for more information.

Expected results:

Command succeeds.

Additional info:

The current deployment (I guess using OSP8-Director) uses the old v2.0 API as the default one as specified by the Keystone service catalog :

[heat-admin@overcloud-controller-0 ~]$ keystone endpoint-get --service compute
+-------------------+-------------------------------------------------------------+
|      Property     | Value                            |
+-------------------+-------------------------------------------------------------+
| compute.publicURL | http://172.20.0.10:8774/v2/eaedf6e2e30840f08a96725eb602b900 |
+-------------------+-------------------------------------------------------------+

Looking at api-paste.ini (the mapping between the REST endpoint and the actual WSGI app), I can confirm that /v2 still points to the legacy v2.0 REST API that doesn't support microversions.

That's very unfortunate that we discover such big gap in the middle of a troubleshooting, because using the legacy API (which is still supported tho) prevents us from getting any feature or bugfix provided for the Liberty timeframe that impacted the REST API. That's not only blocking us from using force-down, but that also prevents us to play with the below :
http://docs.openstack.org/developer/nova/api_microversion_history.html

For sure, I don't see major features but force-down, but that still means we're stuck with an old API.

Not sure how to raise that correctly, but Director obviously lacks from following https://wiki.openstack.org/wiki/ReleaseNotes/Liberty#Upgrade_Notes_2


That said, please keep in mind that depending on the novaclient version you use, force-down could still be not usable. To remember some thread we discussed with Intel :

- novaclient-2.30.1 (currently deployed on OSP8 IIUC) doesn't support remote consoles but should support force-down feature
- novaclient-2.30.2 caps the max acceptable API microversion to 2.5 (see above) which means that we loose support for force-down
- some patches in review plan to fix the gap to raise the accepted microversion to the latest, so we could potentially be able to use the latest client vs. OSP8.


Also:


$ # grab the token
$ KEY=$(keystone token-get | grep id | grep -v _id | \
   sed -e 's/|\s\+id\s\+|\s\+\([a-f0-9]\+\)\s\+|/\1/')
$ # and tenant id
$ TENANT=$(keystone token-get |grep tenant_id | \
   sed -e 's/|\s\+tenant_id\s\+|\s\+\([a-f0-9]\+\)\s\+|/\1/')
$ # call the api
$ curl -g -i -X PUT http://ctrl:8774/v2.1/${TENANT}/os-services/force-down \
   -H "X-Auth-Token: ${KEY}" \
   -H "Accept: application/json" \
   -H "Content-Type: application/json" \
   -H "X-OpenStack-Nova-API-Version: 2.11" \
   -d '{"forced_down": true, "binary": "nova-compute", "host": "cpu1"}'
.1 200 OK
Content-Type: application/json
Content-Length: 76
X-Openstack-Nova-Api-Version: 2.11
Vary: X-OpenStack-Nova-API-Version
X-Compute-Request-Id: req-18a1d0b0-56e1-4986-b7c4-43cb5696f2a2
Date: Tue, 08 Dec 2015 21:31:29 GMT

{"service": {"binary": "nova-compute", "host": "cpu1", "forced_down": true}}
$ 

And as you can see it's works fine. Note, that I have to pass 
X-OpenStack-Nova-API-Version header. Next I've dive into novaclient 
commandline tool - and there I've found the issue. In 
$PYTHONPATH/novaclient/__init__.py (nova client installed via 
dnf/yum/pip install), where $PYTHONPATH is one of a reachable via 
interpreter path (in my case it
was /usr/local/lib/python2.7/dist-packages, since I'm on devstack and
Ubuntu 1404), you can easily spot the API_MAX_VERSION variable, which
have incorrect Nova API microversion set.

Anyway. There are two options. First, local one is to change 
API_MAX_VERSION to api_versions.APIVersion("2.11") on every machine 
which needs to change the state of compute service forcefully. Second, 
more sane is to wait for upstream to be merged, since there already is 
a patchset for this[1].

The analysis for the root cause of changing this value on the first 
place was leading me to the bugfix[2], which corrects the 
incompatibility which was introduced in 2.6 micro version, and the fix 
is simply decrease the default supported version number to… 2.5 :( 
which puts anything introduced after that version to be incompatible 
with the client.

Because of the compatibility reasons, I'm afraid, that support for 
forcing down from the nova client would not be enabled on Liberty 
release.


[1] https://review.openstack.org/#/c/253095/
[2] https://review.openstack.org/#/c/239277/