Bug 1050842

ah, I'm using PKI token

and 
[root@turing ~(keystone_admin)]# neutron --debug subnet-list
Authentication required
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/neutronclient/shell.py", line 517, in run_subcommand
    return run_command(cmd, cmd_parser, sub_argv)
  File "/usr/lib/python2.7/site-packages/neutronclient/shell.py", line 78, in run_command
    return cmd.run(known_args)
  File "/usr/lib/python2.7/site-packages/neutronclient/common/command.py", line 35, in run
    return super(OpenStackCommand, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/display.py", line 84, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/neutronclient/common/command.py", line 41, in take_action
    return self.get_data(parsed_args)
  File "/usr/lib/python2.7/site-packages/neutronclient/neutron/v2_0/__init__.py", line 586, in get_data
    data = self.retrieve_list(parsed_args)
  File "/usr/lib/python2.7/site-packages/neutronclient/neutron/v2_0/__init__.py", line 555, in retrieve_list
    data = self.call_server(neutron_client, search_opts, parsed_args)
  File "/usr/lib/python2.7/site-packages/neutronclient/neutron/v2_0/__init__.py", line 527, in call_server
    data = obj_lister(**search_opts)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 108, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 351, in list_subnets
    **_params)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1197, in list
    for r in self._pagination(collection, path, **params):
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1210, in _pagination
    res = self.get(path, params=params)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1183, in get
    headers=headers, params=params)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1168, in retry_request
    headers=headers, params=params)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1103, in do_request
    resp, replybody = self.httpclient.do_request(action, method, body=body)
  File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 192, in do_request
    self.endpoint_url + url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 156, in _cs_request
    raise exceptions.Unauthorized(message=body)
Unauthorized: Authentication required

Matthias,

It'd also be useful to add any other related snippets in

  - /var/log/keystone.log
  - /var/log/neutron/server/log

keystone log:
tons of messages like 
2014-01-09 10:06:27.081 16281 WARNING keystone.common.utils [-] Deprecated: v2 API is deprecated as of Icehouse in favor of v3 API and may be removed in K.

(I wonder if that's related at all)

and the interesting snippet from neutron server log is posted in above:
That's the log produced, when running neutron subnet-list

the deprecation warning isn't related.

Can you check the auth_token parts of the neutron config file? Specifically have a look to see where the signing_dir is set. This is where the PKI certificates are saved. Make sure that they have been downloaded and is accessible (that would indicate that the auth_url is set correctly).

If it's reproducable and you are willing to hack around with things have a look at keystoneclient/common/cms.py:101 cms_verify function. you can see the openssl command that is generated to verify the PKI token. If you log that command you should be able to run it yourself and maybe see why the process is failing.

[keystone_authtoken]
# auth_host = 127.0.0.1
auth_host = 127.0.0.1
# auth_port = 35357
auth_port = 35357
# auth_protocol = http
auth_protocol = http
# admin_tenant_name = %SERVICE_TENANT_NAME%
admin_tenant_name = services
# admin_user = %SERVICE_USER%
admin_user = neutron
# admin_password = %SERVICE_PASSWORD%
admin_password = a0684557452e407e
# signing_dir = $state_path/keystone-signing
auth_uri=http://127.0.0.1:5000/

setting signing dir = /tmp fixes this issue

from my logs,  /var/lib/neutron/keystone-signing was used earlier, when it didn't work

There's bad value set in neutron-dist.conf:

signing_dir = $state_path/keystone-signing

http://pkgs.fedoraproject.org/cgit/openstack-neutron.git/tree/neutron-dist.conf#n21

This should be removed to let default from keystoneclient/authtoken apply.

I can confirm, removing that lins from neutron-dist.conf makes it work, and I didn't define signing_dir in /etc/neutron/neutron.conf

I have encountered this problem too, when I want to build two controllers for HA.

Tried to reproduce the issue:
- installed fedora 20;
- installed openstack-packstack;
- run packstack --gen-answer-file=answers.txt;
- set CONFIG_QPID_ENABLE_AUTH = y;
- run packstack --answer-file=answers.txt;

- hit bug 1066112, so applied workaround:
> $ sudo touch /var/log/mysqld.log
> $ sudo chown mysql:mysql /var/log/mysqld.log
> $ sudo chcon system_u:object_r:mysqld_log_t:s0 /var/log/mysqld.log
- restarted packstack --answer-file=answers.txt;

- once packstack installation is done, sourced keystonerc_admin file and issued 'neutron subnet-list' with no issues. (empty output, echo $? == 0)

Ihar, 

when using environment variables for authentication: is that the same as using client libraries via keystone token? 

Horizon gets a token from keystone and passes that token to neutron. I *believe*, when providing username and password via env. variables, neutron handles all that auth stuff. That sounds significantly different to me.

The same applies here: neutronclient (which is represented by 'neutron' binary) gets a token from keystone and passes that token to neutron (which is represented by 'neutron-server' binary).

I see the default /etc/neutron/neutron.conf has always had a signing_dir specified. I'm guessing that that was replaced by puppet rewriting neutron.conf ?
So for sane defaults without puppet involved, it seems like we should remove this value from the default neutron.conf also, and provide an equivalent patch upstream?