Bug 1291106 - force_down API not available
Summary: force_down API not available
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-novaclient
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
Target Milestone: ga
: 8.0 (Liberty)
Assignee: Nikola Dipanov
QA Contact: Gabriel Szasz
Depends On:
Blocks: 1185030
TreeView+ depends on / blocked
Reported: 2015-12-14 00:35 UTC by Andrew Beekhof
Modified: 2019-09-09 13:29 UTC (History)
13 users (show)

Fixed In Version: python-novaclient-3.1.0-2.el7ost
Doc Type: Rebase: Bug Fixes and Enhancements
Doc Text:
Clone Of:
Last Closed: 2016-04-07 21:17:52 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:0603 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 8 Enhancement Advisory 2016-04-08 00:53:53 UTC

Description Andrew Beekhof 2015-12-14 00:35:50 UTC
Description of problem:

Instance HA relies in part on the nova.services.force_down() command.
This is not currently available on OSP8.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:

export OS_NO_CACHE=True
export OS_USERNAME=admin
export no_proxy=,,,
export OS_TENANT_NAME=admin
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=
export NOVA_VERSION=2.11
export OS_PASSWORD=...

1. nova service-force-down ${HOST} nova-compute

Actual results:

[root@overcloud-controller-0 heat-admin]# nova service-force-down overcloud-novacompute-1.localdomain nova-compute
No handlers could be found for logger "keystoneclient.auth.identity.generic.base"
ERROR (DiscoveryFailure): Could not determine a suitable URL for the plugin

or, with OS_AUTH_URL=

[root@overcloud-controller-0 heat-admin]# nova service-force-down overcloud-novacompute-1.localdomain nova-compute
usage: nova [--version] [--debug] [--os-cache] [--timings]
            [--os-auth-token OS_AUTH_TOKEN]
            [--os-tenant-name <auth-tenant-name>]
            [--os-tenant-id <auth-tenant-id>] [--os-region-name <region-name>]
            [--os-auth-system <auth-system>] [--service-type <service-type>]
            [--service-name <service-name>]
            [--volume-service-name <volume-service-name>]
            [--os-endpoint-type <endpoint-type>]
            [--os-compute-api-version <compute-api-ver>]
            [--bypass-url <bypass-url>] [--insecure]
            [--os-cacert <ca-certificate>] [--os-cert <certificate>]
            [--os-key <key>] [--timeout <seconds>] [--os-auth-url OS_AUTH_URL]
            [--os-domain-id OS_DOMAIN_ID] [--os-domain-name OS_DOMAIN_NAME]
            [--os-project-id OS_PROJECT_ID]
            [--os-project-name OS_PROJECT_NAME]
            [--os-project-domain-id OS_PROJECT_DOMAIN_ID]
            [--os-project-domain-name OS_PROJECT_DOMAIN_NAME]
            [--os-trust-id OS_TRUST_ID] [--os-user-id OS_USER_ID]
            [--os-user-name OS_USERNAME]
            [--os-user-domain-id OS_USER_DOMAIN_ID]
            [--os-user-domain-name OS_USER_DOMAIN_NAME]
            [--os-password OS_PASSWORD]
            <subcommand> ...
error: argument <subcommand>: invalid choice: u'service-force-down'
Try 'nova help ' for more information.

Expected results:

Command succeeds.

Additional info:

The current deployment (I guess using OSP8-Director) uses the old v2.0 API as the default one as specified by the Keystone service catalog :

[heat-admin@overcloud-controller-0 ~]$ keystone endpoint-get --service compute
|      Property     | Value                            |
| compute.publicURL | |

Looking at api-paste.ini (the mapping between the REST endpoint and the actual WSGI app), I can confirm that /v2 still points to the legacy v2.0 REST API that doesn't support microversions.

That's very unfortunate that we discover such big gap in the middle of a troubleshooting, because using the legacy API (which is still supported tho) prevents us from getting any feature or bugfix provided for the Liberty timeframe that impacted the REST API. That's not only blocking us from using force-down, but that also prevents us to play with the below :

For sure, I don't see major features but force-down, but that still means we're stuck with an old API.

Not sure how to raise that correctly, but Director obviously lacks from following https://wiki.openstack.org/wiki/ReleaseNotes/Liberty#Upgrade_Notes_2

That said, please keep in mind that depending on the novaclient version you use, force-down could still be not usable. To remember some thread we discussed with Intel :

- novaclient-2.30.1 (currently deployed on OSP8 IIUC) doesn't support remote consoles but should support force-down feature
- novaclient-2.30.2 caps the max acceptable API microversion to 2.5 (see above) which means that we loose support for force-down
- some patches in review plan to fix the gap to raise the accepted microversion to the latest, so we could potentially be able to use the latest client vs. OSP8.


$ # grab the token
$ KEY=$(keystone token-get | grep id | grep -v _id | \
   sed -e 's/|\s\+id\s\+|\s\+\([a-f0-9]\+\)\s\+|/\1/')
$ # and tenant id
$ TENANT=$(keystone token-get |grep tenant_id | \
   sed -e 's/|\s\+tenant_id\s\+|\s\+\([a-f0-9]\+\)\s\+|/\1/')
$ # call the api
$ curl -g -i -X PUT http://ctrl:8774/v2.1/${TENANT}/os-services/force-down \
   -H "X-Auth-Token: ${KEY}" \
   -H "Accept: application/json" \
   -H "Content-Type: application/json" \
   -H "X-OpenStack-Nova-API-Version: 2.11" \
   -d '{"forced_down": true, "binary": "nova-compute", "host": "cpu1"}'
.1 200 OK
Content-Type: application/json
Content-Length: 76
X-Openstack-Nova-Api-Version: 2.11
Vary: X-OpenStack-Nova-API-Version
X-Compute-Request-Id: req-18a1d0b0-56e1-4986-b7c4-43cb5696f2a2
Date: Tue, 08 Dec 2015 21:31:29 GMT

{"service": {"binary": "nova-compute", "host": "cpu1", "forced_down": true}}

And as you can see it's works fine. Note, that I have to pass 
X-OpenStack-Nova-API-Version header. Next I've dive into novaclient 
commandline tool - and there I've found the issue. In 
$PYTHONPATH/novaclient/__init__.py (nova client installed via 
dnf/yum/pip install), where $PYTHONPATH is one of a reachable via 
interpreter path (in my case it
was /usr/local/lib/python2.7/dist-packages, since I'm on devstack and
Ubuntu 1404), you can easily spot the API_MAX_VERSION variable, which
have incorrect Nova API microversion set.

Anyway. There are two options. First, local one is to change 
API_MAX_VERSION to api_versions.APIVersion("2.11") on every machine 
which needs to change the state of compute service forcefully. Second, 
more sane is to wait for upstream to be merged, since there already is 
a patchset for this[1].

The analysis for the root cause of changing this value on the first 
place was leading me to the bugfix[2], which corrects the 
incompatibility which was introduced in 2.6 micro version, and the fix 
is simply decrease the default supported version number to… 2.5 :( 
which puts anything introduced after that version to be incompatible 
with the client.

Because of the compatibility reasons, I'm afraid, that support for 
forcing down from the nova client would not be enabled on Liberty 

[1] https://review.openstack.org/#/c/253095/
[2] https://review.openstack.org/#/c/239277/

Comment 2 Nikola Dipanov 2015-12-21 10:45:31 UTC
Latest upstream release that supports mark host down is 3.1.0 out at this point so we should include it in OSP 8

Comment 7 errata-xmlrpc 2016-04-07 21:17:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.