Bug 1659078

Summary: Octavia with TLS everywhere infrared deployment fails
Product: Red Hat OpenStack Reporter: Bernard Cafarelli <bcafarel>
Component: openstack-tripleoAssignee: Brent Eagles <beagles>
Status: CLOSED WORKSFORME QA Contact: Arik Chernetsky <achernet>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 14.0 (Rocky)CC: abregman, bcafarel, cgoncalves, hrybacki, ihrachys, jagee, josorior, lpeer, majopela, mburns
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-02-21 14:44:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/var/lib/mistral/overcloud/ansible.log
none
overcloud_install.log none

Description Bernard Cafarelli 2018-12-13 14:14:25 UTC 
Deploying OSP 14 with Octavia and TLS everywhere fails when trying to upload the amphora image file.

This was tested with infrared( adding a freeipa node to topology, and --tls-everywhere True), with this change in to use new template files:
https://review.gerrithub.io/c/redhat-openstack/infrared/+/435998 (as suggested in https://bugzilla.redhat.com/show_bug.cgi?id=1608391 )

Deployment fails at "Configure octavia on overcloud": steps here need to contact the overcloud (OS_AUTH_URL), but fail to resolve overcloud.redhat.local (will upload overcloud install and mistral ansible log)
On overcloud nodes, this entry is present /etc/hosts in the HEAT_HOSTS_START/END section, but not on undercloud (or mistral container)

Without TLS everywhere, this is not an issue as we use the IP address to connect to the overcloud.

I suspect the issue is in tripleo (and not infrared), as these entries are not added by IR

Failed to discover available identity versions when contacting https://overcloud.redhat.local:13000/v3. Attempting to parse version from URL.\\nUnable to establish connection to https://overcloud.redhat.local:13000/v3/auth/tokens: HTTPSConnectionPool(host='overcloud.redhat.local', port=13000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0a6b0f0110>: Failed to establish a new connection: [Errno -2] Name or service not known',))\", \"stderr_lines\": [\"Failed to discover available identity versions when contacting https://overcloud.redhat.local:13000/v3. Attempting to parse version from URL.\", \"Unable to establish connection to https://overcloud.redhat.local:13000/v3/auth/tokens: HTTPSConnectionPool(host='overcloud.redhat.local', port=13000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.
VerifiedHTTPSConnection object at 0x7f0a6b0f0110>: Failed to establish a new connection: [Errno -2] Name or service not known',))\
Comment 1 Bernard Cafarelli 2018-12-13 14:14:53 UTC 
Created attachment 1514066 [details]
/var/lib/mistral/overcloud/ansible.log
Comment 2 Bernard Cafarelli 2018-12-13 14:15:23 UTC 
Created attachment 1514067 [details]
overcloud_install.log
Comment 3 Bernard Cafarelli 2018-12-13 14:17:52 UTC 
Maybe relevant to this hosts issue, after deployment fails, I created an overcloudrc file:
. stackrc; openstack overcloud credentials overcloud

It has:
export OS_AUTH_URL=https://overcloud.redhat.local:13000/v3

and trying to use this file for CLI commands fails:
(overcloud) [stack@undercloud-0 ~]$ openstack server list
Failed to discover available identity versions when contacting https://overcloud.redhat.local:13000/v3. Attempting to parse version from URL.
Unable to establish connection to https://overcloud.redhat.local:13000/v3/auth/tokens: HTTPSConnectionPool(host='overcloud.redhat.local', port=13000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f9e28361710>: Failed to establish a new connection: [Errno -2] Nom ou service inconnu',))

So I guess the hosts are updated later on TLS everywhere deployments? (which may not be enough for Octavia steps, if container uses another hosts file)
Comment 4 Carlos Goncalves 2018-12-13 18:16:26 UTC 
Moving to DFG:Security as this regards TLS everywhere.

For reference, the image upload part here mentioned is run by config-download on step 5. Specifically, https://github.com/openstack/tripleo-common/blob/master/playbooks/octavia-files.yaml and https://github.com/openstack/tripleo-common/blob/master/playbooks/roles/octavia-undercloud/tasks/image_mgmt.yml
Comment 5 Jeremy Agee 2019-01-17 20:35:19 UTC 
(In reply to Bernard Cafarelli from comment #3)
> Maybe relevant to this hosts issue, after deployment fails, I created an
> overcloudrc file:
> . stackrc; openstack overcloud credentials overcloud
> 
> It has:
> export OS_AUTH_URL=https://overcloud.redhat.local:13000/v3
> 
> and trying to use this file for CLI commands fails:
> (overcloud) [stack@undercloud-0 ~]$ openstack server list
> Failed to discover available identity versions when contacting
> https://overcloud.redhat.local:13000/v3. Attempting to parse version from
> URL.
> Unable to establish connection to
> https://overcloud.redhat.local:13000/v3/auth/tokens:
> HTTPSConnectionPool(host='overcloud.redhat.local', port=13000): Max retries
> exceeded with url: /v3/auth/tokens (Caused by
> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
> 0x7f9e28361710>: Failed to establish a new connection: [Errno -2] Nom ou
> service inconnu',))
> 
> So I guess the hosts are updated later on TLS everywhere deployments? (which
> may not be enough for Octavia steps, if container uses another hosts file)

Hi Bernard it looks like you might want to try and use haproxy-public-tls-certmonger.yaml for your overcloud as well as a public_vip mapping.  You will need to add the following dns entrys on the ipa server. adjust the info lab.local to redhat.local for your setup. from the hypervisor you can ssh to the freeipa-0 vm. feel free to let reach out to me if you have questions. 
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/integrate_with_identity_service/idm-novajoin#configure_dns_entries_for_novajoin
Comment 6 Harry Rybacki 2019-02-01 16:48:29 UTC 
Adding NEEDINFO to raise visibility of comment#5 to Bernard.
Comment 8 Juan Antonio Osorio 2019-02-15 14:57:38 UTC 
Did Jeremy's comment address the issue?
Comment 9 Bernard Cafarelli 2019-02-15 15:21:29 UTC 
Hi Juan, yes this outlined procedure sounds like the way to fix this issue, we should also have support for it in infrared