Bug 1659078 - Octavia with TLS everywhere infrared deployment fails
Summary: Octavia with TLS everywhere infrared deployment fails
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Brent Eagles
QA Contact: Arik Chernetsky
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-13 14:14 UTC by Bernard Cafarelli
Modified: 2019-09-10 14:12 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-02-21 14:44:54 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
/var/lib/mistral/overcloud/ansible.log (2.56 MB, text/plain)
2018-12-13 14:14 UTC, Bernard Cafarelli
no flags Details
overcloud_install.log (2.58 MB, text/plain)
2018-12-13 14:15 UTC, Bernard Cafarelli
no flags Details

Description Bernard Cafarelli 2018-12-13 14:14:25 UTC
Deploying OSP 14 with Octavia and TLS everywhere fails when trying to upload the amphora image file.

This was tested with infrared( adding a freeipa node to topology, and --tls-everywhere True), with this change in to use new template files:
https://review.gerrithub.io/c/redhat-openstack/infrared/+/435998 (as suggested in https://bugzilla.redhat.com/show_bug.cgi?id=1608391 )

Deployment fails at "Configure octavia on overcloud": steps here need to contact the overcloud (OS_AUTH_URL), but fail to resolve overcloud.redhat.local (will upload overcloud install and mistral ansible log)
On overcloud nodes, this entry is present /etc/hosts in the HEAT_HOSTS_START/END section, but not on undercloud (or mistral container)

Without TLS everywhere, this is not an issue as we use the IP address to connect to the overcloud.

I suspect the issue is in tripleo (and not infrared), as these entries are not added by IR

Failed to discover available identity versions when contacting https://overcloud.redhat.local:13000/v3. Attempting to parse version from URL.\\nUnable to establish connection to https://overcloud.redhat.local:13000/v3/auth/tokens: HTTPSConnectionPool(host='overcloud.redhat.local', port=13000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0a6b0f0110>: Failed to establish a new connection: [Errno -2] Name or service not known',))\", \"stderr_lines\": [\"Failed to discover available identity versions when contacting https://overcloud.redhat.local:13000/v3. Attempting to parse version from URL.\", \"Unable to establish connection to https://overcloud.redhat.local:13000/v3/auth/tokens: HTTPSConnectionPool(host='overcloud.redhat.local', port=13000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.
VerifiedHTTPSConnection object at 0x7f0a6b0f0110>: Failed to establish a new connection: [Errno -2] Name or service not known',))\

Comment 1 Bernard Cafarelli 2018-12-13 14:14:53 UTC
Created attachment 1514066 [details]
/var/lib/mistral/overcloud/ansible.log

Comment 2 Bernard Cafarelli 2018-12-13 14:15:23 UTC
Created attachment 1514067 [details]
overcloud_install.log

Comment 3 Bernard Cafarelli 2018-12-13 14:17:52 UTC
Maybe relevant to this hosts issue, after deployment fails, I created an overcloudrc file:
. stackrc; openstack overcloud credentials overcloud

It has:
export OS_AUTH_URL=https://overcloud.redhat.local:13000/v3

and trying to use this file for CLI commands fails:
(overcloud) [stack@undercloud-0 ~]$ openstack server list
Failed to discover available identity versions when contacting https://overcloud.redhat.local:13000/v3. Attempting to parse version from URL.
Unable to establish connection to https://overcloud.redhat.local:13000/v3/auth/tokens: HTTPSConnectionPool(host='overcloud.redhat.local', port=13000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f9e28361710>: Failed to establish a new connection: [Errno -2] Nom ou service inconnu',))

So I guess the hosts are updated later on TLS everywhere deployments? (which may not be enough for Octavia steps, if container uses another hosts file)

Comment 4 Carlos Goncalves 2018-12-13 18:16:26 UTC
Moving to DFG:Security as this regards TLS everywhere.

For reference, the image upload part here mentioned is run by config-download on step 5. Specifically, https://github.com/openstack/tripleo-common/blob/master/playbooks/octavia-files.yaml and https://github.com/openstack/tripleo-common/blob/master/playbooks/roles/octavia-undercloud/tasks/image_mgmt.yml

Comment 5 Jeremy Agee 2019-01-17 20:35:19 UTC
(In reply to Bernard Cafarelli from comment #3)
> Maybe relevant to this hosts issue, after deployment fails, I created an
> overcloudrc file:
> . stackrc; openstack overcloud credentials overcloud
> 
> It has:
> export OS_AUTH_URL=https://overcloud.redhat.local:13000/v3
> 
> and trying to use this file for CLI commands fails:
> (overcloud) [stack@undercloud-0 ~]$ openstack server list
> Failed to discover available identity versions when contacting
> https://overcloud.redhat.local:13000/v3. Attempting to parse version from
> URL.
> Unable to establish connection to
> https://overcloud.redhat.local:13000/v3/auth/tokens:
> HTTPSConnectionPool(host='overcloud.redhat.local', port=13000): Max retries
> exceeded with url: /v3/auth/tokens (Caused by
> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
> 0x7f9e28361710>: Failed to establish a new connection: [Errno -2] Nom ou
> service inconnu',))
> 
> So I guess the hosts are updated later on TLS everywhere deployments? (which
> may not be enough for Octavia steps, if container uses another hosts file)

Hi Bernard it looks like you might want to try and use haproxy-public-tls-certmonger.yaml for your overcloud as well as a public_vip mapping.  You will need to add the following dns entrys on the ipa server. adjust the info lab.local to redhat.local for your setup. from the hypervisor you can ssh to the freeipa-0 vm. feel free to let reach out to me if you have questions. 
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/integrate_with_identity_service/idm-novajoin#configure_dns_entries_for_novajoin

Comment 6 Harry Rybacki 2019-02-01 16:48:29 UTC
Adding NEEDINFO to raise visibility of comment#5 to Bernard.

Comment 8 Juan Antonio Osorio 2019-02-15 14:57:38 UTC
Did Jeremy's comment address the issue?

Comment 9 Bernard Cafarelli 2019-02-15 15:21:29 UTC
Hi Juan, yes this outlined procedure sounds like the way to fix this issue, we should also have support for it in infrared


Note You need to log in before you can comment on or make changes to this bug.