Bug 1058291

I just approved the linked review which will i think provide only a section of this bug. 

Review: https://review.openstack.org/#/c/67474/ will make it so that the discover command works like every other keystone command and will now respond to the --insecure and --timeout flags. 

As the cause of this bug is in keystoneclient it will not follow the openstack release cycle - however i am not aware of when the next release is expected. 

These flags will need to be added to the whatever is calling the keystoneclient script. They can be added before the client is released, and they will simply be ignored until. 

From looking at https://github.com/redhat-openstack/openstack-keystone/blob/master/openstack-keystone.service I can't see where the initial call is happening though. Am i looking at too new a version? Do you know where the discover call is coming from?

(In reply to Jamie Lennox from comment #2)
> I just approved the linked review which will i think provide only a section
> of this bug. 
> 
> Review: https://review.openstack.org/#/c/67474/ will make it so that the
> discover command works like every other keystone command and will now
> respond to the --insecure and --timeout flags. 
>
> As the cause of this bug is in keystoneclient it will not follow the
> openstack release cycle - however i am not aware of when the next release is
> expected. 

While the issue is with the client this does make it appear that the overall openstack-keystone service has hung at startup, would that change how we address this issue?

> These flags will need to be added to the whatever is calling the
> keystoneclient script. They can be added before the client is released, and
> they will simply be ignored until.
>
> From looking at
> https://github.com/redhat-openstack/openstack-keystone/blob/master/openstack-
> keystone.service I can't see where the initial call is happening though. Am
> i looking at too new a version? Do you know where the discover call is
> coming from?

Ah yes, my apologies for not clearly stating that this was a RHOS 4 issue on RHEL 6.5 hosts. Thus the above systemd service file doesn't apply here. The SysVinit file calls and then waits for 'keystone discover' to return causing the service to appear to hang here.

It won't change the default behaviour initially, but it will mean that we can add --insecure and --timeout=5 to the discover call in the startup script (we can add them now they'll just be ignored).

--insecure is to not verify a certificate
--timeout is how long before we give up on making a request. 

passing those should have the same effect as the changed parameters in the diff from comment 1.

Targeting for A2.  We should cherry-pick the upstream keystoneclient patch.  A patch for the init script needs to be proposed.

(In reply to Nathan Kinder from comment #7)
> Targeting for A2.  We should cherry-pick the upstream keystoneclient patch. 

Changing component for this part.

> A patch for the init script needs to be proposed.

This part is obsoleted by bug 1036515 comment 21 change in openstack-keystone, also targeting A2.

Keystone client released with discover fix, i think it was 0.5.1 (might have been 0.5.0). Need the init script timeout fix still.

(In reply to Jamie Lennox from comment #11)
> Keystone client released with discover fix, i think it was 0.5.1 (might have
> been 0.5.0). Need the init script timeout fix still.

We can't rebase to >= 0.4.2 as it breaks puppet-keystone module which still uses deprecated --endpoint option which was removed in 0.4.2.

So puppet-keystone should be updated to use --os-endpoint instead of --endpoint which has been supported for a while (though i can't say exactly when --os-endpoint was added).

I'm not sure if this has been filed elsewhere so just putting needinfo to rob as i think that puppet-keystone is his department.

The switch to --os-endpoint is under review now at https://review.openstack.org/#/c/73970/

IMHO --insecure should be avoided.

(In reply to Rob Crittenden from comment #19)
> The switch to --os-endpoint is under review now at
> https://review.openstack.org/#/c/73970/

Filed bug 1066627 for openstack-puppet-modules

The patch provided in this bug is not present in the Havana A2 release. I still checked if the service starts when configured with SSL - and it started successfully so somehow it is fixed another way.

Hovever, I tried a simple "keystone user-list" command and it hanged indefinitely. When I pressed Ctrl+C I got this exception:

  File "/usr/bin/keystone", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.6/site-packages/keystoneclient/shell.py", line 490, in main
    OpenStackIdentityShell().main(sys.argv[1:])
  File "/usr/lib/python2.6/site-packages/keystoneclient/shell.py", line 433, in main
    timeout=args.timeout)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 139, in __init__
    self.authenticate()
  File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 468, in authenticate
    resp, body = self.get_raw_token_from_identity_service(**kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 162, in get_raw_token_from_identity_service
    token=token)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 197, in _base_authN
    resp, body = self.request(url, 'POST', body=params, headers=headers)
  File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 610, in request
    **request_kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 112, in request
    **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 174, in send
    timeout=timeout
  File "/usr/lib/python2.6/site-packages/urllib3/connectionpool.py", line 427, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python2.6/site-packages/urllib3/connectionpool.py", line 289, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 349, in _read_status
    line = self.fp.readline()
  File "/usr/lib64/python2.6/socket.py", line 433, in readline
    data = recv(1)
KeyboardInterrupt


I tried to change OS_AUTH_URL in keystonerc_admin to look for https://... and it didn't work at all - it complained that it can't establish a connection.

Should I fail this bug? I need an explanation on how this bug was fixed if it wasn't fixed using our patch, and how to succeed with the client when keystone runs with ssl enabled.

By the way, to configure keystone with ssl, I uncommented the following lines in keystone.conf:
[ssl]
enable = True
certfile = /etc/keystone/pki/certs/ssl_cert.pem
keyfile = /etc/keystone/pki/private/ssl_key.pem
ca_certs = /etc/keystone/pki/certs/cacert.pem
ca_key = /etc/keystone/pki/private/cakey.pem

And then I ran the command:
keystone-manage ssl_setup --keystone-user keystone --keystone-group keystone

.. and restarted the service. Is that the right procedure?

(In reply to Udi from comment #22)
> The patch provided in this bug is not present in the Havana A2 release. I
> still checked if the service starts when configured with SSL - and it
> started successfully so somehow it is fixed another way.
> 
...
> 
> Should I fail this bug? I need an explanation on how this bug was fixed if
> it wasn't fixed using our patch, and how to succeed with the client when
> keystone runs with ssl enabled.

I believe that Alan fixed it in the Keystone init script as mentioned here:

    https://bugzilla.redhat.com/show_bug.cgi?id=1036515#c26

(In reply to Udi from comment #22)
> The patch provided in this bug is not present in the Havana A2 release. I
> still checked if the service starts when configured with SSL - and it
> started successfully so somehow it is fixed another way.
> 
> Hovever, I tried a simple "keystone user-list" command and it hanged
> indefinitely.

This hang sounds normal for a non-SSL client that tries to connect to a SSL enabled server.  That makes me think that you have the server properly configured for SSL.

> 
> I tried to change OS_AUTH_URL in keystonerc_admin to look for https://...
> and it didn't work at all - it complained that it can't establish a
> connection.

You should be specifying "https://".  Would you supply the exact OS_AUTH_URL that you are using along with the exact error that it reports?

It's quite possible that the certificate is not being validated properly due to a hostname mismatch between the CN in the certificate subject DN and the hostname that you are using in OS_AUTH_URL.  The "keystone" client has a "--insecure" option that skips certificate trust checking, but I'm not sure it that would skip a hostname mismatch.  It would be a better test to make sure we have a valid SSL setup with correct hostnames.

Alan's fix does not concern keystone, but nova. So, I still don't see exactly how the bug was fixed but anyways the service starts.

I use the following OS_AUTH_URL:
export OS_AUTH_URL=https://127.0.0.1:35357/v2.0/

I get this error:
keystone user-list
Authorization Failed: Unable to establish connection to https://127.0.0.1:35357/v2.0/tokens

I also tried replacing 127.0.0.1 with the real IP (actually that was the default) and it returns the same error. With the --insecure option I get:
Unable to establish connection to http://10.8.0.50:35357/v2.0/users

Should the bug be marked as verified even though I can't authenticate with the client, and can't authenticate with the API (curl) either?

(In reply to Udi from comment #25)
> I also tried replacing 127.0.0.1 with the real IP (actually that was the
> default) and it returns the same error. With the --insecure option I get:
> Unable to establish connection to http://10.8.0.50:35357/v2.0/users

You should change identity endpoint in service catalog to use https.

Verified in: python-keystoneclient-0.4.1-4.el6ost.noarch

Service started with SSL enabled. Changed the endpoint urls to https://... and was able to list the catalog with 'keystone --insecure catalog'.

apevec, can you explain why the --insecure flag is needed? Thanks.

> apevec, can you explain why the --insecure flag is needed?

  --insecure            Explicitly allow keystoneclient to perform "insecure"
                        TLS (https) requests. The server's certificate will
                        not be verified against any certificate authorities.
                        This option should be used with caution.

Steps to reproduce were:
1) uncomment the lines from keystone.conf under the [ssl] section
2) run keystone-manage ssl_setup --keystone-user keystone --keystone-group keystone

^^^ here is self-signed certificate created, hence --insecure

3) update endpoint.url (http: -> https:) in keystone database
4) restart the openstack-keystone service
5) change OS_AUTH_URL to https: in keystonerc_admin and re-source it

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0213.html