Sorry, I must have missed this one somehow.
What a mess. I wonder if introspection can help us figure out which form we need to use.

Changing the Nova API in our product such that other products that interact with it break is a regression.

After talking with danpb and artom, the commit in 330516543b052b29c9663139086d635c7a27b301 is what caused the problem.

novaclient expects kwargs as input.  fence_compute is sending positional args instead, which is why the issue occurred.

The order changed, but if fence_compute used kwargs instead of positional ones, it should work.

https://github.com/openstack/python-novaclient/commit/330516543b052b29c9663139086d635c7a27b301

This commit sorted a bunch of parameters, so anything not using kwargs broke.

Artom discovered that the above may not have been the culprit.

https://github.com/openstack/python-novaclient/commit/f98b8470de2e0225befc5a39ef20ed5e4f7880c1

Wait, project_name and project_id are different things, aren't they?

project_id is going a the tenant ID, not the tenant name - project_id still exists, and is in the same positional parameter.

If we want to use 'tenant name' to mean tenant name, that's fine, but I would think project_id is going to want the ID.

fence_compute seems to call the client API with --tenant-name in the positional parameter for project_id in stable/newton and master for python-novaclient.

Also on Ocata, tenant_name and tenant_id are translated to project_name and project_id, respectively:

_check_arguments(kwargs, "Ocata", "tenant_id", right_name="project_id")
_check_arguments(kwargs, "Ocata", "tenant_name", right_name="project_name")

On an *OSP10* instance:

* Positional args works:

>>> from novaclient import client as nc
>>> c = nc.Client('2.11', 'admin', '[deleted]', 'admin', 'http://172.16.23.10:5000/v2.0)
>>> print c
<novaclient.v2.client.Client object at 0x2dee950>
>>> c.hypervisors.list()
[<Hypervisor: 2>, <Hypervisor: 5>]

* Changing to use explicit parameters works:

>>> from novaclient import client as nc
>>> c = nc.Client('2.11', username='admin', password='[deleted]', project_id='admin', auth_url='http://172.16.23.10:5000/v2.0')
>>> print c
<novaclient.v2.client.Client object at 0x18e4950>
>>> c.hypervisors.list()
[<Hypervisor: 2>, <Hypervisor: 5>]

* Changing to use project_name on OSP10 breaks:

>>> c = nc.Client('2.11', username='admin', password='[deleted]', project_name='admin', auth_url='http://172.16.23.10:5000/v2.0')
...
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 529, in _extract_service_catalog
    raise exceptions.AuthorizationFailure()
novaclient.exceptions.AuthorizationFailure

* Using the database ID for 'admin' resulted in a 401/unauthorized.

>>> c = nc.Client('2.11', username='admin', password='[deleted]', project_id='a5f0dc09587d45ecaac0cf8ba88487df', auth_url='http://172.16.23.10:5000/v2.0')
>>> c.hypervisors.list()
...
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 425, in request
    raise exceptions.from_response(resp, body, url, method)
novaclient.exceptions.Unauthorized: The request you have made requires authentication. (HTTP 401)

So, my points above in comment #9 comment #10 comment #11 seem to be incorrect for OSP10.

I'll have to retest (or someone else can) with OSP11.  Project_id should still work on OSP11, though.

On OSP11:

>>> from novaclient import client as nc

*without* kwargs:

>>> c = nc.Client('2.11', 'admin', '[deleted]', 'admin', 'http://192.168.122.158:5000/v2.0')
>>> c.hypervisors.list()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 402, in substitution
    return methods[-1].func(obj, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/hypervisors.py", line 55, in list
    return self._list_base(detailed=detailed)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/hypervisors.py", line 48, in _list_base
    return self._list(path, 'hypervisors')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 254, in _list
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 217, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 74, in request
    **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 374, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 142, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 491, in request
    auth_headers = self.get_auth_headers(auth)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 818, in get_auth_headers
    return auth.get_headers(self, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers
    token = self.get_token(session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 90, in get_token
    return self.get_access(session).auth_token
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref
    return self._plugin.get_auth_ref(session, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
    authenticated=False, log=False)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 766, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 655, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-0e15aaed-9f89-4fdc-a3a2-00c7416a4c43)


*with* kwargs using project_id:

>>> c = nc.Client('2.11', username='admin', password='[deleted]', project_id='admin', auth_url='http://192.168.122.158:5000/v2.0')
>>> c.hypervisors.list()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 402, in substitution
    return methods[-1].func(obj, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/hypervisors.py", line 55, in list
    return self._list_base(detailed=detailed)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/hypervisors.py", line 48, in _list_base
    return self._list(path, 'hypervisors')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 254, in _list
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 217, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 74, in request
    **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 374, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 142, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 491, in request
    auth_headers = self.get_auth_headers(auth)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 818, in get_auth_headers
    return auth.get_headers(self, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers
    token = self.get_token(session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 90, in get_token
    return self.get_access(session).auth_token
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref
    return self._plugin.get_auth_ref(session, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
    authenticated=False, log=False)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 766, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 655, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-d5131362-4d74-4666-9d5d-baabb1be5a04)
>>> 


*with* kwargs using project_name:

>>> c = nc.Client('2.11', username='admin', password='15a77b09a5de45db', project_name='admin', auth_url='http://192.168.122.158:5000/v2.0')
>>> c.hypervisors.list()
[<Hypervisor: 1>]

It looks like project_id now really wants a project_id:

>>> from novaclient import client as nc
>>> c = nc.Client('2.11', 'admin', '15a77b09a5de45db', 'f8c2777b02a6485ea296c47a1d2b5ac4', 'http://192.168.122.158:5000/v2.0')
>>> c.hypervisors.list()
[<Hypervisor: 1>]
>>> c2 = nc.Client('2.11', username='admin', password='15a77b09a5de45db', project_id='f8c2777b02a6485ea296c47a1d2b5ac4', auth_url='http://192.168.122.158:5000/v2.0')
>>> c2.hypervisors.list()
[<Hypervisor: 1>]

... which actually makes a lot more sense than the previous version.

https://review.openstack.org/#/c/350106/

^ Diana Clarke pointed this out.

The patch adds project_name (to mean "human-readable name") and changes the meaning of project_id from "human readable name" to "ID from database".

Marian: Could you please try the following patch which converts everything to named arguments instead of positional?

diff --git a/fence/agents/compute/fence_compute.py b/fence/agents/compute/fence_compute.py
index 0a238b6..f426013 100644
--- a/fence/agents/compute/fence_compute.py
+++ b/fence/agents/compute/fence_compute.py
@@ -310,11 +310,11 @@ def create_nova_connection(options):
 
        versions = [ "2.11", "2" ]
        for version in versions:
-               nova = client.Client(version,
-                                    options["--username"],
-                                    options["--password"],
-                                    options["--tenant-name"],
-                                    options["--auth-url"],
+               nova = client.Client(api_version=version,
+                                    username=options["--username"],
+                                    api_key=options["--password"],
+                                    auth_url=options["--auth-url"],
+                                    tenant_name=options["--tenant-name"],
                                     insecure=options["--insecure"],
                                     region_name=options["--region-name"],
                                     endpoint_type=options["--endpoint-type"],

Sorry, one change too many. Version is the only non-named argument so:

diff --git a/fence/agents/compute/fence_compute.py b/fence/agents/compute/fence_compute.py
index 0a238b6..9a4adf0 100644
--- a/fence/agents/compute/fence_compute.py
+++ b/fence/agents/compute/fence_compute.py
@@ -311,10 +311,10 @@ def create_nova_connection(options):
        versions = [ "2.11", "2" ]
        for version in versions:
                nova = client.Client(version,
-                                    options["--username"],
-                                    options["--password"],
-                                    options["--tenant-name"],
-                                    options["--auth-url"],
+                                    username=options["--username"],
+                                    api_key=options["--password"],
+                                    auth_url=options["--auth-url"],
+                                    tenant_name=options["--tenant-name"],
                                     insecure=options["--insecure"],
                                     region_name=options["--region-name"],
                                     endpoint_type=options["--endpoint-type"],

(In reply to Andrew Beekhof from comment #20)
> Sorry, one change too many. Version is the only non-named argument so:
> 
> diff --git a/fence/agents/compute/fence_compute.py
> b/fence/agents/compute/fence_compute.py
> index 0a238b6..9a4adf0 100644
> --- a/fence/agents/compute/fence_compute.py
> +++ b/fence/agents/compute/fence_compute.py
> @@ -311,10 +311,10 @@ def create_nova_connection(options):
>         versions = [ "2.11", "2" ]
>         for version in versions:
>                 nova = client.Client(version,
> -                                    options["--username"],
> -                                    options["--password"],
> -                                    options["--tenant-name"],
> -                                    options["--auth-url"],
> +                                    username=options["--username"],
> +                                    api_key=options["--password"],
> +                                    auth_url=options["--auth-url"],
> +                                    tenant_name=options["--tenant-name"],
>                                      insecure=options["--insecure"],
>                                      region_name=options["--region-name"],
>                                     
> endpoint_type=options["--endpoint-type"],

I tested all the IHA stuff following the steps here [1] which applied smoothly.
I can confirm that the non-named argument approach solves the issue.
There's a problem once the fenced machine comes up again, since it always (== on each reboot) hangs while registering ebtables and needs an additional manual reset.
This problem does not seem to be related to the bug we're describing here, I'll investigate and (in case) open a new bug.

[1] https://github.com/redhat-openstack/tripleo-quickstart-utils/blob/master/roles/instance-ha/tasks/main.yml

(In reply to Raoul Scarazzini from comment #21)
> (In reply to Andrew Beekhof from comment #20)
> > Sorry, one change too many. Version is the only non-named argument so:
> > 
> > diff --git a/fence/agents/compute/fence_compute.py
> > b/fence/agents/compute/fence_compute.py
> > index 0a238b6..9a4adf0 100644
> > --- a/fence/agents/compute/fence_compute.py
> > +++ b/fence/agents/compute/fence_compute.py
> > @@ -311,10 +311,10 @@ def create_nova_connection(options):
> >         versions = [ "2.11", "2" ]
> >         for version in versions:
> >                 nova = client.Client(version,
> > -                                    options["--username"],
> > -                                    options["--password"],
> > -                                    options["--tenant-name"],
> > -                                    options["--auth-url"],
> > +                                    username=options["--username"],
> > +                                    api_key=options["--password"],
> > +                                    auth_url=options["--auth-url"],
> > +                                    tenant_name=options["--tenant-name"],
> >                                      insecure=options["--insecure"],
> >                                      region_name=options["--region-name"],
> >                                     
> > endpoint_type=options["--endpoint-type"],
> 
> I tested all the IHA stuff following the steps here [1] which applied
> smoothly.
> I can confirm that the non-named argument approach solves the issue.

We're converting TO named arguments. Did you mean non-positional?

> There's a problem once the fenced machine comes up again, since it always
> (== on each reboot) hangs while registering ebtables and needs an additional
> manual reset.

I don't understand this paragraph.

> This problem does not seem to be related to the bug we're describing here,
> I'll investigate and (in case) open a new bug.
> 
> [1]
> https://github.com/redhat-openstack/tripleo-quickstart-utils/blob/master/
> roles/instance-ha/tasks/main.yml

(In reply to Andrew Beekhof from comment #22)
[...]
> We're converting TO named arguments. Did you mean non-positional?

Yes, my mistake, by the way I used the same code you suggested on comment #20.

> > There's a problem once the fenced machine comes up again, since it always
> > (== on each reboot) hangs while registering ebtables and needs an additional
> > manual reset.
> I don't understand this paragraph.

Every time I test IHA, by hanging a compute node, instances living on this node are correctly moved to another compute, then the original compute gets fenced. At this point, while booting up again after being fenced, the machine hangs after activating network phase (I see the ebtables on video as the last operation, but this could not be related).
As I said I'm investigating this because it happens every time.

The project_name alone is not unique, you need to specify a project_domain_name to make it unique, so you are not forced to use project_id.

It is the same with user names, you need to specify a user_domain_name next to the username to make it unique.

Many code parts still have the fall back  logic for the 'Default' named domain or to the 'default' domain_id, when they suspect a not domain aware client.

(In reply to Andrew Beekhof from comment #20)
> Sorry, one change too many. Version is the only non-named argument so:
> 
> diff --git a/fence/agents/compute/fence_compute.py
> b/fence/agents/compute/fence_compute.py
> index 0a238b6..9a4adf0 100644
> --- a/fence/agents/compute/fence_compute.py
> +++ b/fence/agents/compute/fence_compute.py
> @@ -311,10 +311,10 @@ def create_nova_connection(options):
>         versions = [ "2.11", "2" ]
>         for version in versions:
>                 nova = client.Client(version,
> -                                    options["--username"],
> -                                    options["--password"],
> -                                    options["--tenant-name"],
> -                                    options["--auth-url"],
> +                                    username=options["--username"],
> +                                    api_key=options["--password"],
> +                                    auth_url=options["--auth-url"],
> +                                    tenant_name=options["--tenant-name"],
>                                      insecure=options["--insecure"],
>                                      region_name=options["--region-name"],
>                                     
> endpoint_type=options["--endpoint-type"],

It does not work for me on RHOSP10 (and older) unless I change the name of argument "tenant_name" to "project_id". Note that to tenant_name/project_id argument I am supplying the value of OS_TENANT_NAME variable from generated overcloudrc.

On the other hand On RHOSP11 I get following warnings:
UserWarning: The 'api_key' argument is deprecated in Ocata and its use may result in errors in future releases. Use 'password' instead.
UserWarning: The 'tenant_name' argument is deprecated in Ocata and its use may result in errors in future releases. Use 'project_name' instead.
But It eventually works.

(In reply to Marian Krcmarik from comment #25)
> (In reply to Andrew Beekhof from comment #20)
> > Sorry, one change too many. Version is the only non-named argument so:
> > 
> > diff --git a/fence/agents/compute/fence_compute.py
> > b/fence/agents/compute/fence_compute.py
> > index 0a238b6..9a4adf0 100644
> > --- a/fence/agents/compute/fence_compute.py
> > +++ b/fence/agents/compute/fence_compute.py
> > @@ -311,10 +311,10 @@ def create_nova_connection(options):
> >         versions = [ "2.11", "2" ]
> >         for version in versions:
> >                 nova = client.Client(version,
> > -                                    options["--username"],
> > -                                    options["--password"],
> > -                                    options["--tenant-name"],
> > -                                    options["--auth-url"],
> > +                                    username=options["--username"],
> > +                                    api_key=options["--password"],
> > +                                    auth_url=options["--auth-url"],
> > +                                    tenant_name=options["--tenant-name"],
> >                                      insecure=options["--insecure"],
> >                                      region_name=options["--region-name"],
> >                                     
> > endpoint_type=options["--endpoint-type"],
> 
> It does not work for me on RHOSP10 (and older) unless I change the name of
> argument "tenant_name" to "project_id". 

Because nova doesn't recognise an argument with that name? 
Or because 'project_domain_name' is not provided?

> Note that to tenant_name/project_id
> argument I am supplying the value of OS_TENANT_NAME variable from generated
> overcloudrc.

I don't think 'project_id' is correct.
Based on the OSP11 message, I would expect that it should be at least 'project_name'.

> 
> On the other hand On RHOSP11 I get following warnings:
> UserWarning: The 'api_key' argument is deprecated in Ocata and its use may
> result in errors in future releases. Use 'password' instead.

Does 10 understand 'password' though? If not, we'll have to live with that one.

> UserWarning: The 'tenant_name' argument is deprecated in Ocata and its use
> may result in errors in future releases. Use 'project_name' instead.
> But It eventually works.

The above code part is not keystone v3 and domain friendly, but the v2 auth is still supported, so it might not be critical for 11, but fence_compute tool definitely will need to support more auth parameter in the future.

You can check the `openstack help` for the frequently passed arguments, at least the ones winch contains domain should be passable here as well.

FYI: You might be interested in using sessions (single thread):
https://docs.openstack.org/developer/python-keystoneclient/using-sessions.html

The keystone's auto guessing auth-type/auth-version might lead to miss leading conclusions when you just change one parameter, it also does not have fully consistent behavior across releases.

(In reply to Andrew Beekhof from comment #26)
> (In reply to Marian Krcmarik from comment #25)
> > (In reply to Andrew Beekhof from comment #20)
> > > Sorry, one change too many. Version is the only non-named argument so:
> > > 
> > > diff --git a/fence/agents/compute/fence_compute.py
> > > b/fence/agents/compute/fence_compute.py
> > > index 0a238b6..9a4adf0 100644
> > > --- a/fence/agents/compute/fence_compute.py
> > > +++ b/fence/agents/compute/fence_compute.py
> > > @@ -311,10 +311,10 @@ def create_nova_connection(options):
> > >         versions = [ "2.11", "2" ]
> > >         for version in versions:
> > >                 nova = client.Client(version,
> > > -                                    options["--username"],
> > > -                                    options["--password"],
> > > -                                    options["--tenant-name"],
> > > -                                    options["--auth-url"],
> > > +                                    username=options["--username"],
> > > +                                    api_key=options["--password"],
> > > +                                    auth_url=options["--auth-url"],
> > > +                                    tenant_name=options["--tenant-name"],
> > >                                      insecure=options["--insecure"],
> > >                                      region_name=options["--region-name"],
> > >                                     
> > > endpoint_type=options["--endpoint-type"],
> > 
> > It does not work for me on RHOSP10 (and older) unless I change the name of
> > argument "tenant_name" to "project_id". 
> 
> Because nova doesn't recognise an argument with that name? 
> Or because 'project_domain_name' is not provided?

I tried something like:
nova = client.Client("2.11",username="admin", password="ktH23dVcXTUrZvveNuBUHtqpz", auth_url="http://[2620:52:0:13b8:5054:ff:fe3e:4]:5000/v2.0", tenant_name="admin", user_domain_name="Default", project_domain_name="Default")
But did not help, maybe we need to try to create keystone session first as suggested by Attila, I dunno...
> 
> > Note that to tenant_name/project_id
> > argument I am supplying the value of OS_TENANT_NAME variable from generated
> > overcloudrc.
> 
> I don't think 'project_id' is correct.
> Based on the OSP11 message, I would expect that it should be at least
> 'project_name'.
> 
> > 
> > On the other hand On RHOSP11 I get following warnings:
> > UserWarning: The 'api_key' argument is deprecated in Ocata and its use may
> > result in errors in future releases. Use 'password' instead.
> 
> Does 10 understand 'password' though? If not, we'll have to live with that
> one.
password seems to be working on older releases
> 
> > UserWarning: The 'tenant_name' argument is deprecated in Ocata and its use
> > may result in errors in future releases. Use 'project_name' instead.
> > But It eventually works.

(In reply to Marian Krcmarik from comment #28)

> I tried something like:
> nova = client.Client("2.11",username="admin",
> password="ktH23dVcXTUrZvveNuBUHtqpz",
> auth_url="http://[2620:52:0:13b8:5054:ff:fe3e:4]:5000/v2.0",
> tenant_name="admin", user_domain_name="Default",
> project_domain_name="Default")
> But did not help, maybe we need to try to create keystone session first as
> suggested by Attila, I dunno...

Well it depends on how far back keystone v3 goes, but either way, we still need to figure out which parameter names and values to use.

PM Approved. Business justification: Currently blocking CI phase 2 automation testing in OSP 11, which is blocking the path to beta, RC and GA. Patch will ensure that customers do not experience service disruptions when OSP 11 is released for production use.

Attila: The problem here is that OSP8 can't take named arguments and OSP11 only takes them. 

Marian:  This patch seems to do ok (with tenant_name as a named parameter)

>>> nova = client.Client("2", username="admin", password="MyQyFERp9tkfgej6abEkpPfqA", auth_url="http://10.0.0.110:5000/v2.0", tenant_name="admin", insecure="False", region_name="", endpoint_type="internalURL", http_log_debug="False")
>>> nova.hypervisors.list()
[<Hypervisor: 3>, <Hypervisor: 6>]



diff --git a/fence/agents/compute/fence_compute.py b/fence/agents/compute/fence_compute.py
index 0a238b6..c529de5 100644
--- a/fence/agents/compute/fence_compute.py
+++ b/fence/agents/compute/fence_compute.py
@@ -4,6 +4,7 @@ import sys
 import time
 import atexit
 import logging
+import inspect
 import requests.exceptions
 
 sys.path.append("@FENCEAGENTSLIBDIR@")
@@ -310,15 +311,42 @@ def create_nova_connection(options):
 
        versions = [ "2.11", "2" ]
        for version in versions:
-               nova = client.Client(version,
-                                    options["--username"],
-                                    options["--password"],
-                                    options["--tenant-name"],
-                                    options["--auth-url"],
-                                    insecure=options["--insecure"],
-                                    region_name=options["--region-name"],
-                                    endpoint_type=options["--endpoint-type"],
-                                    http_log_debug=options.has_key("--verbose"))
+                clientargs = inspect.getargspec(client.Client).args
+
+                # Some versions of Openstack prior to Ocata only
+                # supported positional arguments for username,
+                # password and tenant.
+                #
+                # Versions since Ocata only support named arguments.
+                #
+                # So we need to use introspection to figure out how to
+                # create a Nova client.
+                #
+                # Happy days
+                #
+                if len(clientargs) > 1:
+                        # OSP < 11
+                       nova = client.Client(version,
+                                            options["--username"],
+                                            options["--password"],
+                                            options["--tenant-name"],
+                                            auth_url=options["--auth-url"],
+                                            insecure=options["--insecure"],
+                                            region_name=options["--region-name"],
+                                            endpoint_type=options["--endpoint-type"],
+                                            http_log_debug=options.has_key("--verbose"))
+                else:
+                        # OSP >= 11
+                       nova = client.Client(version,
+                                            username=options["--username"],
+                                            password=options["--password"],
+                                            tenant_name=options["--tenant-name"],
+                                            auth_url=options["--auth-url"],
+                                            insecure=options["--insecure"],
+                                            region_name=options["--region-name"],
+                                            endpoint_type=options["--endpoint-type"],
+                                            http_log_debug=options.has_key("--verbose"))
+
                try:
                        nova.hypervisors.list()
                        return
@@ -329,7 +357,7 @@ def create_nova_connection(options):
                except Exception as e:
                        logging.warning("Nova connection failed. %s: %s" % (e.__class__.__name__, e))
                        
-       fail_usage("Couldn't obtain a supported connection to nova, tried: %s" % repr(versions))
+       logging.warning("Couldn't obtain a supported connection to nova, tried: %s\n" % repr(versions))
 
 def define_new_opts():
        all_opt["endpoint-type"] = {
@@ -417,7 +445,7 @@ def main():
        global override_status
        atexit.register(atexit_handler)
 
-       device_opt = ["login", "passwd", "tenant-name", "auth-url", "fabric_fencing", "on_target",
+       device_opt = ["login", "passwd", "tenant-name", "auth-url", "fabric_fencing", 
                "no_login", "no_password", "port", "domain", "no-shared-storage", "endpoint-type",
                "record-only", "instance-filtering", "insecure", "region-name"]
        define_new_opts()

Slightly improved patch (that now makes sense conceptually too)

[11:23 AM] beekhof@fedora ~/Development/sources/fence-agents/upstream ☺ # git diff
diff --git a/fence/agents/compute/fence_compute.py b/fence/agents/compute/fence_compute.py
index 0a238b6..4b229b0 100644
--- a/fence/agents/compute/fence_compute.py
+++ b/fence/agents/compute/fence_compute.py
@@ -4,6 +4,7 @@ import sys
 import time
 import atexit
 import logging
+import inspect
 import requests.exceptions
 
 sys.path.append("@FENCEAGENTSLIBDIR@")
@@ -310,15 +311,46 @@ def create_nova_connection(options):
 
        versions = [ "2.11", "2" ]
        for version in versions:
-               nova = client.Client(version,
-                                    options["--username"],
-                                    options["--password"],
-                                    options["--tenant-name"],
-                                    options["--auth-url"],
-                                    insecure=options["--insecure"],
-                                    region_name=options["--region-name"],
-                                    endpoint_type=options["--endpoint-type"],
-                                    http_log_debug=options.has_key("--verbose"))
+                clientargs = inspect.getargspec(client.Client).varargs
+
+                # Some versions of Openstack prior to Ocata only
+                # supported positional arguments for username,
+                # password and tenant.
+                #
+                # Versions since Ocata only support named arguments.
+                #
+                # So we need to use introspection to figure out how to
+                # create a Nova client.
+                #
+                # Happy days
+                #
+                if clientargs:
+                        # OSP < 11
+                        # ArgSpec(args=['version', 'username', 'password', 'project_id', 'auth_url'],
+                        #         varargs=None,
+                        #         keywords='kwargs', defaults=(None, None, None, None))
+                       nova = client.Client(version,
+                                            options["--username"],
+                                            options["--password"],
+                                            options["--tenant-name"],
+                                            options["--auth-url"],
+                                            insecure=options["--insecure"],
+                                            region_name=options["--region-name"],
+                                            endpoint_type=options["--endpoint-type"],
+                                            http_log_debug=options.has_key("--verbose"))
+                else:
+                        # OSP >= 11
+                        # ArgSpec(args=['version'], varargs='args', keywords='kwargs', defaults=None)
+                       nova = client.Client(version,
+                                            username=options["--username"],
+                                            password=options["--password"],
+                                            tenant_name=options["--tenant-name"],
+                                            auth_url=options["--auth-url"],
+                                            insecure=options["--insecure"],
+                                            region_name=options["--region-name"],
+                                            endpoint_type=options["--endpoint-type"],
+                                            http_log_debug=options.has_key("--verbose"))
+
                try:
                        nova.hypervisors.list()
                        return
@@ -329,7 +361,7 @@ def create_nova_connection(options):
                except Exception as e:
                        logging.warning("Nova connection failed. %s: %s" % (e.__class__.__name__, e))
                        
-       fail_usage("Couldn't obtain a supported connection to nova, tried: %s" % repr(versions))
+       logging.warning("Couldn't obtain a supported connection to nova, tried: %s\n" % repr(versions))
 
 def define_new_opts():
        all_opt["endpoint-type"] = {
@@ -417,7 +449,7 @@ def main():
        global override_status
        atexit.register(atexit_handler)
 
-       device_opt = ["login", "passwd", "tenant-name", "auth-url", "fabric_fencing", "on_target",
+       device_opt = ["login", "passwd", "tenant-name", "auth-url", "fabric_fencing", 
                "no_login", "no_password", "port", "domain", "no-shared-storage", "endpoint-type",
                "record-only", "instance-filtering", "insecure", "region-name"]
        define_new_opts()
 : git : ✗ master@d53b047 Merge branch 'vuntz-fix-domain'
[11:23 AM] beekhof@fedora ~/Development/sources/fence-agents/upstream ☺ #

Created attachment 1267767 [details]
fence-agents fixes

Code Verified.

Thanks for dealing with the whitespace problems while I was on PTO!

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1874