Bug 1291106 - force_down API not available
force_down API not available
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-novaclient (Show other bugs)
8.0 (Liberty)
Unspecified Unspecified
urgent Severity urgent
: ga
: 8.0 (Liberty)
Assigned To: Nikola Dipanov
Gabriel Szasz
: Rebase, Triaged
Depends On:
Blocks: 1185030
  Show dependency treegraph
Reported: 2015-12-13 19:35 EST by Andrew Beekhof
Modified: 2016-04-07 17:17 EDT (History)
14 users (show)

See Also:
Fixed In Version: python-novaclient-3.1.0-2.el7ost
Doc Type: Rebase: Bug Fixes and Enhancements
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2016-04-07 17:17:52 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Andrew Beekhof 2015-12-13 19:35:50 EST
Description of problem:

Instance HA relies in part on the nova.services.force_down() command.
This is not currently available on OSP8.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:

export OS_NO_CACHE=True
export OS_USERNAME=admin
export no_proxy=,,,
export OS_TENANT_NAME=admin
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=
export NOVA_VERSION=2.11
export OS_PASSWORD=...

1. nova service-force-down ${HOST} nova-compute

Actual results:

[root@overcloud-controller-0 heat-admin]# nova service-force-down overcloud-novacompute-1.localdomain nova-compute
No handlers could be found for logger "keystoneclient.auth.identity.generic.base"
ERROR (DiscoveryFailure): Could not determine a suitable URL for the plugin

or, with OS_AUTH_URL=

[root@overcloud-controller-0 heat-admin]# nova service-force-down overcloud-novacompute-1.localdomain nova-compute
usage: nova [--version] [--debug] [--os-cache] [--timings]
            [--os-auth-token OS_AUTH_TOKEN]
            [--os-tenant-name <auth-tenant-name>]
            [--os-tenant-id <auth-tenant-id>] [--os-region-name <region-name>]
            [--os-auth-system <auth-system>] [--service-type <service-type>]
            [--service-name <service-name>]
            [--volume-service-name <volume-service-name>]
            [--os-endpoint-type <endpoint-type>]
            [--os-compute-api-version <compute-api-ver>]
            [--bypass-url <bypass-url>] [--insecure]
            [--os-cacert <ca-certificate>] [--os-cert <certificate>]
            [--os-key <key>] [--timeout <seconds>] [--os-auth-url OS_AUTH_URL]
            [--os-domain-id OS_DOMAIN_ID] [--os-domain-name OS_DOMAIN_NAME]
            [--os-project-id OS_PROJECT_ID]
            [--os-project-name OS_PROJECT_NAME]
            [--os-project-domain-id OS_PROJECT_DOMAIN_ID]
            [--os-project-domain-name OS_PROJECT_DOMAIN_NAME]
            [--os-trust-id OS_TRUST_ID] [--os-user-id OS_USER_ID]
            [--os-user-name OS_USERNAME]
            [--os-user-domain-id OS_USER_DOMAIN_ID]
            [--os-user-domain-name OS_USER_DOMAIN_NAME]
            [--os-password OS_PASSWORD]
            <subcommand> ...
error: argument <subcommand>: invalid choice: u'service-force-down'
Try 'nova help ' for more information.

Expected results:

Command succeeds.

Additional info:

The current deployment (I guess using OSP8-Director) uses the old v2.0 API as the default one as specified by the Keystone service catalog :

[heat-admin@overcloud-controller-0 ~]$ keystone endpoint-get --service compute
|      Property     | Value                            |
| compute.publicURL | |

Looking at api-paste.ini (the mapping between the REST endpoint and the actual WSGI app), I can confirm that /v2 still points to the legacy v2.0 REST API that doesn't support microversions.

That's very unfortunate that we discover such big gap in the middle of a troubleshooting, because using the legacy API (which is still supported tho) prevents us from getting any feature or bugfix provided for the Liberty timeframe that impacted the REST API. That's not only blocking us from using force-down, but that also prevents us to play with the below :

For sure, I don't see major features but force-down, but that still means we're stuck with an old API.

Not sure how to raise that correctly, but Director obviously lacks from following https://wiki.openstack.org/wiki/ReleaseNotes/Liberty#Upgrade_Notes_2

That said, please keep in mind that depending on the novaclient version you use, force-down could still be not usable. To remember some thread we discussed with Intel :

- novaclient-2.30.1 (currently deployed on OSP8 IIUC) doesn't support remote consoles but should support force-down feature
- novaclient-2.30.2 caps the max acceptable API microversion to 2.5 (see above) which means that we loose support for force-down
- some patches in review plan to fix the gap to raise the accepted microversion to the latest, so we could potentially be able to use the latest client vs. OSP8.


$ # grab the token
$ KEY=$(keystone token-get | grep id | grep -v _id | \
   sed -e 's/|\s\+id\s\+|\s\+\([a-f0-9]\+\)\s\+|/\1/')
$ # and tenant id
$ TENANT=$(keystone token-get |grep tenant_id | \
   sed -e 's/|\s\+tenant_id\s\+|\s\+\([a-f0-9]\+\)\s\+|/\1/')
$ # call the api
$ curl -g -i -X PUT http://ctrl:8774/v2.1/${TENANT}/os-services/force-down \
   -H "X-Auth-Token: ${KEY}" \
   -H "Accept: application/json" \
   -H "Content-Type: application/json" \
   -H "X-OpenStack-Nova-API-Version: 2.11" \
   -d '{"forced_down": true, "binary": "nova-compute", "host": "cpu1"}'
.1 200 OK
Content-Type: application/json
Content-Length: 76
X-Openstack-Nova-Api-Version: 2.11
Vary: X-OpenStack-Nova-API-Version
X-Compute-Request-Id: req-18a1d0b0-56e1-4986-b7c4-43cb5696f2a2
Date: Tue, 08 Dec 2015 21:31:29 GMT

{"service": {"binary": "nova-compute", "host": "cpu1", "forced_down": true}}

And as you can see it's works fine. Note, that I have to pass 
X-OpenStack-Nova-API-Version header. Next I've dive into novaclient 
commandline tool - and there I've found the issue. In 
$PYTHONPATH/novaclient/__init__.py (nova client installed via 
dnf/yum/pip install), where $PYTHONPATH is one of a reachable via 
interpreter path (in my case it
was /usr/local/lib/python2.7/dist-packages, since I'm on devstack and
Ubuntu 1404), you can easily spot the API_MAX_VERSION variable, which
have incorrect Nova API microversion set.

Anyway. There are two options. First, local one is to change 
API_MAX_VERSION to api_versions.APIVersion("2.11") on every machine 
which needs to change the state of compute service forcefully. Second, 
more sane is to wait for upstream to be merged, since there already is 
a patchset for this[1].

The analysis for the root cause of changing this value on the first 
place was leading me to the bugfix[2], which corrects the 
incompatibility which was introduced in 2.6 micro version, and the fix 
is simply decrease the default supported version number to… 2.5 :( 
which puts anything introduced after that version to be incompatible 
with the client.

Because of the compatibility reasons, I'm afraid, that support for 
forcing down from the nova client would not be enabled on Liberty 

[1] https://review.openstack.org/#/c/253095/
[2] https://review.openstack.org/#/c/239277/
Comment 2 Nikola Dipanov 2015-12-21 05:45:31 EST
Latest upstream release that supports mark host down is 3.1.0 out at this point so we should include it in OSP 8
Comment 7 errata-xmlrpc 2016-04-07 17:17:52 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.