Bug 1395392 - When the Undercloud is configured with SSL the TripleO UI won't allow logging in failing with "Connection to Keystone is not available"
Summary: When the Undercloud is configured with SSL the TripleO UI won't allow logging...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: instack-undercloud
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: rc
: 11.0 (Ocata)
Assignee: Dan Trainor
QA Contact: Ola Pavlenko
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-15 21:43 UTC by Ramon Acedo
Modified: 2020-02-14 18:09 UTC (History)
13 users (show)

Fixed In Version: instack-undercloud-6.0.0-2.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-17 19:46:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 441478 0 None None None 2017-04-13 16:36:49 UTC
Launchpad 1668775 0 None None None 2017-03-29 22:21:59 UTC
Red Hat Product Errata RHEA-2017:1245 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 11.0 Bug Fix and Enhancement Advisory 2017-05-17 23:01:50 UTC

Description Ramon Acedo 2016-11-15 21:43:04 UTC
A fresh install of the Undercloud with SSL enabled won't allow logging in to TripleO UI when accessing failing with "Connection to Keystone is not available" when clicking on the "Log In" button.

On another deployment without SSL this works without any issues.

Comment 1 Florian Fuchs 2016-11-16 11:08:39 UTC
Are you using Firefox? If so, does this workaround help?

https://bugzilla.redhat.com/show_bug.cgi?id=1392627#c5

Comment 2 Ramon Acedo 2016-11-16 14:40:48 UTC
This happened in both, Chrome and Firefox from MacOS X. I didn't see any exception related to SSL certificates, it's just a message from the UI saying "Connection to Keystone is not available".

Comment 3 Benjamin Schmaus 2016-12-16 01:31:25 UTC
I seem to be seeing this behaviour as well in my lab setup.

Comment 4 Sebastián Greco 2016-12-30 07:50:31 UTC
Hi,

Just trying Newton in my lab. Same error happens to me using 3 or 4 web browsers including Chrome on Windows. This is my undercloud.conf file for the installation.

[DEFAULT]

image_path = .
undercloud_hostname = os-dir01.essi.lab
local_ip = 172.16.1.74/24
network_gateway = 172.16.1.74
undercloud_public_vip = 172.16.1.72
undercloud_admin_vip = 172.16.1.73
#undercloud_service_certificate =
generate_service_certificate = true
certificate_generation_ca = local
#service_principal =
local_interface = ens224
local_mtu = 1500
network_cidr = 172.16.1.0/24
masquerade_network = 172.16.1.0/24
dhcp_start = 172.16.1.51
dhcp_end = 172.16.1.60
#hieradata_override =
#net_config_override =
inspection_interface = br-ctlplane
inspection_iprange = 172.16.1.61,172.16.1.65
inspection_extras = true
inspection_runbench = true
inspection_enable_uefi = true
undercloud_debug = true
enable_tempest = true
enable_mistral = true
enable_zaqar = true
enable_telemetry = true
enable_ui = true
enable_validations = true
ipxe_enabled = true
store_events = true
scheduler_max_attempts = 30
clean_nodes = true

Comment 5 Sebastián Greco 2016-12-30 08:31:32 UTC
(In reply to Ramon Acedo from comment #0)
> A fresh install of the Undercloud with SSL enabled won't allow logging in to
> TripleO UI when accessing failing with "Connection to Keystone is not
> available" when clicking on the "Log In" button.
> 
> On another deployment without SSL this works without any issues.

Hi,

For what I can read and understand from the Installation guide, only way to use UI is when having SSL/TLS enabled.

"
enable_ui
Defines Whether to install the director’s web UI. This allows you to perform overcloud
planning and deployments through a graphical web interface. For more information, see
Chapter 6, Configuring Basic Overcloud Requirements with the Web UI. Note that the UI is
only available with SSL/TLS enabled using either the
undercloud_service_certificate or generate_service_certificate.
"

Am I missing something perhaps? Any feedback and/or workaround regarding this problem is most appreciated.

Regards,
Seb

Comment 6 Sebastián Greco 2017-01-04 14:24:29 UTC
Hi,

I am able to reproduce the error everytime. And now, I'm also able to fix it (somehow).

What I do to fix it is to navigate with my web browser to the keystone IP:ports

After doing that, I am able to log in. I reproduced the installation 2 times, facing this error. Both of the times the error was fixed doing that.

Hope it helps someone.

Regards,
Seb

Comment 7 Dan Trainor 2017-01-04 18:38:37 UTC
(In reply to Sebastián Greco from comment #6)
> Hi,
> 
> I am able to reproduce the error everytime. And now, I'm also able to fix it
> (somehow).
> 
> What I do to fix it is to navigate with my web browser to the keystone
> IP:ports
> 
> After doing that, I am able to log in. I reproduced the installation 2
> times, facing this error. Both of the times the error was fixed doing that.
> 
> Hope it helps someone.
> 
> Regards,
> Seb

Can you confirm that you are using Chrome and no other browser?  To elaborate on Florian's comment, Chrome is the only supported browser at this time[0].  There is a set of patches that will be provided upstream to modify the way that Director UI talks to backend services, allowing any browser to be used.  I've pinged Docs team to confirm this is documented; I wasn't able to find that documentation in our Product notes.

If you have made any modifications to /var/www/openstack-tripleo-ui/dist/tripleo_ui_config.js, can you provide this file please?


---

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1392627

Comment 8 Sebastián Greco 2017-01-05 07:20:49 UTC
Hi,

I'm using Chrome v55 64-bit over a Windows 10 machine.

I did not made any modifications to "/var/www/openstack-tripleo-ui/dist/tripleo_ui_config.js" at least not intentionally. I just followed this guide step by step.
"https://access.redhat.com/webassets/avalon/d/Red_Hat_OpenStack_Platform-10-Director_Installation_and_Usage-en-US/Red_Hat_OpenStack_Platform-10-Director_Installation_and_Usage-en-US.pdf"

If you still need that file I will be glad to send it to you (i'm not in the office right now). Even better, if you need access to my lab, we can arrange that too. I have snpashots of the vms almost before any major step, so reproducing the exact conditions is easy.

My kung-fu is not too strong, but please do let me know if there's anything I could do to help.

Regards,
Seb

Comment 9 Benjamin Schmaus 2017-01-24 02:31:38 UTC
I wonder if the problem here is that the services defined in tripleo_ui_config.js assume that the deployment network IP on director is routable and accessible.

For example, some customers do not have a routable deployment network, it is only layer 2.

In my home lab, I too have a non routable or accessible deployment network.  So when I access the UI of my director node I am hitting IP 192.168.0.7 whereas my director deployment network br-ctl IP is on 192.168.3.201.

If I edit the .js file and change the keystone port to 192.168.0.7 instead of 192.168.3.201, it does get around the keystone error.  However I then hit the issue of Mistral not being accessible because it is listening on 192.168.3.201 but not on 192.168.0.7.

I think for the director node it might make sense if all the GUI components were set to listen on all IP's of the host.

Comment 10 Benjamin Schmaus 2017-01-24 02:51:27 UTC
If I make the deployment network routable, I have no issues when I hit the deployment IP of the director node for the UI.

Not all customers have that privilege though.

Comment 11 Moacir 2017-01-30 12:42:50 UTC
Same problem here.

I noticed that I open the TripleO GUI using, in my case, 192.168.100.2 that is my defined undercloud_public_vip. However if we do a netstat -tunap | grep keystone, we see that keystone is running only on IP address 192.168.100.1, that in my case is my defined local_ip.

Comment 12 Moacir 2017-01-30 16:08:18 UTC
Ok, I guess I found my problem... If I use Google Chrome, it works. The version I have installed is Version 56.0.2924.76 (64-bit).

I hope it helps.

Comment 14 Vincent S. Cojot 2017-02-21 03:38:09 UTC
Hi everyone,
Ok, I had run into the same issue (Connection Error, Connection to Keystone is not available) and I sorted it out. This makes me a little worried about deployments in the enterprise.

Here's how the interaction with the UI works from the client:
1) client connects to undercloud_public_vip on tcp/443 (https)
2) because of the design of the tripleo-ui, client then attempts to connect to undercloud_public_vip on tcp/13000 (keystone).
3) other steps follow..

Here's the same connection when a proxy is defined on the desktop.
1) client connects to the proxy and requests a connection to undercloud_public_vip on tcp/443 (https). This works and the proxy handles the connection on the client's behalf.
2) client then connects to proxy again and requests connection on undercloud_public_vip on tcp/443 (https). In most case, this will get defined because 13000 isn't a whitelisted port in most enterprise proxies. whitelisted ports usually include 80, 443 and a few others.
On my squid proxy, it resulted in the following message logged in the proxy's log:
1487634438.611      0 10.0.128.254 TCP_DENIED/403 4098 CONNECT 10.0.128.42:13000 - HIER_NONE/- text/html
1487634438.611      0 10.0.128.254 TCP_DENIED/403 4098 CONNECT 10.0.128.42:13000 - HIER_NONE/- text/html
1487634438.642      0 10.0.128.254 TCP_DENIED/403 4098 CONNECT 10.0.128.42:13000 

This wouldn't be such a big problem if it weren't for two items of importance:
1) on Linux, OSX and Windows. Google Chrome proxy settings are usually handled by the local IT and users usually cannot change that setting.
On my Linux desktop, I was able to work around the issue by starting chrome like this: "google-chrome-stable --no-proxy-server"

2) it is quite easy to change the proxy settings in firefox but firefox is currently unusable for tripleo-ui usage.

Bottom line, if you got to the BZ because of the keystone connection error message, check your Proxy settings.

Perhaps the tripleo UI could proxy the connections to the different backends/services and feed the results back to the client on port tcp/443 (https).

Thanks,

Comment 15 Vincent S. Cojot 2017-02-21 03:40:18 UTC
Found a small typo, the 2nd 2) should read as follows:
2) client then connects to proxy again and requests connection on undercloud_public_vip on tcp/13000 (keystone)

Comment 16 Moacir 2017-02-21 08:49:15 UTC
I guess this problem only happens because the browser does not recognize the server's certificate. What if the CA certificate were available on the main page so we could download and install it? Would this "fix" the problem?

Comment 17 Dan Trainor 2017-03-29 22:21:59 UTC
Revisiting this, we had a few changes that ultimately had the effect of solving this.

First off, the SSL certificate that we used on the front-end, generated by Certmonger, was creating a certificate with no Extended Key Usage (EKU) attributes (namely,  id-kp-serverAuth and id-kp-clientAuth).  Some browsers (Chrome, for example) do not ensure that these EKUs exist, while others (Firefox) do.  We've changed our deployments to ensure that these EKUs are applied to the certificate that Certmonger generates for us[0].

Secondly, we no longer connect directly to each service endpoint[1].  Instead of entering each service endpoint through a specific port that HAProxy listens for, we now request these services through HTTPS using a '/servicename' part appended to the URL.  An example for the Keystone configuration in tripleo_ui_config.js is as follows:

  'keystone': 'https://1.2.3.4:443/keystone/v2.0',

This allows the connection to be terminated by HAProxy over HTTPS as usual, but then forwards the connection to a new configuration residing in Apache.  Once the connection gets to Apache, it is passed using mod_proxy's ProxyPass and ProxyPassReverse to the specific service endpoint on the backend, routed using the request's '/servicename'.

We still require that the UI be accessed from the "front" using SSL, over port 443.  However, one can access the UI over plain HTTP by accessing the Undercloud's "back" IP address and the port 3000 (the port that the Apache UI VirtualHost container listens on).  The "back" IP address is, by default, 192.168.24.1.

I hope this provides some clarification to this outstanding bug.

---

[0] https://bugs.launchpad.net/tripleo/+bug/1668775
[1] https://blueprints.launchpad.net/tripleo/+spec/proxy-undercloud-api-services

Comment 19 Ola Pavlenko 2017-04-26 16:16:06 UTC
Verified: openstack-tripleo-ui-3.1.0-9.el7ost.noarch
On firefox and chrome, virtual and baremetal

Comment 21 errata-xmlrpc 2017-05-17 19:46:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245


Note You need to log in before you can comment on or make changes to this bug.