+++ This bug was initially created as a clone of Bug #1050842 +++

Description of problem:

I installed via packstack in fedora 20.
After sourcing keystonerc_admin, neutron still complains about missing authentication.


Version-Release number of selected component (if applicable):
rpm -q openstack-neutron
openstack-neutron-2014.1-0.1.b1.fc21.noarch


How reproducible:
100%

Steps to Reproduce:
1. # . ./keystonerc_admin 
2. nova image-list
+----+------+--------+--------+
| ID | Name | Status | Server |
+----+------+--------+--------+
+----+------+--------+--------+

3. # neutron subnet-list
Authentication required


Actual results:
Authentication required

Expected results:
List of subnets

Additional info:

I think it may be connected with the following issue from neutron-server log:
2014-01-09 09:37:51.247 2315 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 127.0.0.1
2014-01-09 09:37:51.317 2315 WARNING keystoneclient.middleware.auth_token [-] Verify error: Command 'openssl' returned non-zero exit status 4
2014-01-09 09:37:51.318 2315 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token 0fc5a67eeb09cb154c04f68255a622c7
2014-01-09 09:37:51.318 2315 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-01-09 09:37:51.320 2315 INFO neutron.wsgi [-] 127.0.0.1 - - [09/Jan/2014 09:37:51] "GET /v2.0/subnets.json HTTP/1.1" 401 192 0.077199

rpm -q openssl
openssl-1.0.1e-36.fc20.x86_64

--- Additional comment from Matthias Runge on 2014-01-09 04:02:13 EST ---

ah, I'm using PKI token

and 
[root@turing ~(keystone_admin)]# neutron --debug subnet-list
Authentication required
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/neutronclient/shell.py", line 517, in run_subcommand
    return run_command(cmd, cmd_parser, sub_argv)
  File "/usr/lib/python2.7/site-packages/neutronclient/shell.py", line 78, in run_command
    return cmd.run(known_args)
  File "/usr/lib/python2.7/site-packages/neutronclient/common/command.py", line 35, in run
    return super(OpenStackCommand, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/display.py", line 84, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/neutronclient/common/command.py", line 41, in take_action
    return self.get_data(parsed_args)
  File "/usr/lib/python2.7/site-packages/neutronclient/neutron/v2_0/__init__.py", line 586, in get_data
    data = self.retrieve_list(parsed_args)
  File "/usr/lib/python2.7/site-packages/neutronclient/neutron/v2_0/__init__.py", line 555, in retrieve_list
    data = self.call_server(neutron_client, search_opts, parsed_args)
  File "/usr/lib/python2.7/site-packages/neutronclient/neutron/v2_0/__init__.py", line 527, in call_server
    data = obj_lister(**search_opts)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 108, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 351, in list_subnets
    **_params)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1197, in list
    for r in self._pagination(collection, path, **params):
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1210, in _pagination
    res = self.get(path, params=params)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1183, in get
    headers=headers, params=params)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1168, in retry_request
    headers=headers, params=params)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1103, in do_request
    resp, replybody = self.httpclient.do_request(action, method, body=body)
  File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 192, in do_request
    self.endpoint_url + url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 156, in _cs_request
    raise exceptions.Unauthorized(message=body)
Unauthorized: Authentication required

--- Additional comment from Kashyap Chamarthy on 2014-01-09 04:06:07 EST ---

Matthias,

It'd also be useful to add any other related snippets in

  - /var/log/keystone.log
  - /var/log/neutron/server/log

--- Additional comment from Matthias Runge on 2014-01-09 04:10:32 EST ---

keystone log:
tons of messages like 
2014-01-09 10:06:27.081 16281 WARNING keystone.common.utils [-] Deprecated: v2 API is deprecated as of Icehouse in favor of v3 API and may be removed in K.

(I wonder if that's related at all)

and the interesting snippet from neutron server log is posted in above:
That's the log produced, when running neutron subnet-list

--- Additional comment from Jamie Lennox on 2014-01-09 06:43:07 EST ---

the deprecation warning isn't related.

Can you check the auth_token parts of the neutron config file? Specifically have a look to see where the signing_dir is set. This is where the PKI certificates are saved. Make sure that they have been downloaded and is accessible (that would indicate that the auth_url is set correctly).

If it's reproducable and you are willing to hack around with things have a look at keystoneclient/common/cms.py:101 cms_verify function. you can see the openssl command that is generated to verify the PKI token. If you log that command you should be able to run it yourself and maybe see why the process is failing.

--- Additional comment from Matthias Runge on 2014-01-09 06:52:31 EST ---

[keystone_authtoken]
# auth_host = 127.0.0.1
auth_host = 127.0.0.1
# auth_port = 35357
auth_port = 35357
# auth_protocol = http
auth_protocol = http
# admin_tenant_name = %SERVICE_TENANT_NAME%
admin_tenant_name = services
# admin_user = %SERVICE_USER%
admin_user = neutron
# admin_password = %SERVICE_PASSWORD%
admin_password = a0684557452e407e
# signing_dir = $state_path/keystone-signing
auth_uri=http://127.0.0.1:5000/

--- Additional comment from Matthias Runge on 2014-01-09 07:01:08 EST ---

setting signing dir = /tmp fixes this issue

from my logs,  /var/lib/neutron/keystone-signing was used earlier, when it didn't work

--- Additional comment from Alan Pevec on 2014-01-09 07:36:08 EST ---

There's bad value set in neutron-dist.conf:

signing_dir = $state_path/keystone-signing

http://pkgs.fedoraproject.org/cgit/openstack-neutron.git/tree/neutron-dist.conf#n21

This should be removed to let default from keystoneclient/authtoken apply.

--- Additional comment from Matthias Runge on 2014-01-09 07:39:52 EST ---

I can confirm, removing that lins from neutron-dist.conf makes it work, and I didn't define signing_dir in /etc/neutron/neutron.conf