Bug 1943500

Summary: openshift installer fails immediately failed to fetch Install Config
Product: OpenShift Container Platform Reporter: Martin André <m.andre>
Component: InstallerAssignee: Martin André <m.andre>
Installer sub component: OpenShift on OpenStack QA Contact: Jon Uriarte <juriarte>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: urgent CC: igreen, mbooth
Version: 4.7Keywords: Triaged
Target Milestone: ---   
Target Release: 4.7.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: A change in gophercloud/utils introduced a custom HTTP client to make use of the self-signed certificate specified in the clouds.yaml file. This change however removed all the settings that came with the DefaultTransport, including handling of proxy environment variables and default timeouts. Consequence: Installation that uses both self-signed certificat and proxy fail. Fix: Resolve the issue in gophercloud/utils by ensuring the custom HTTP client inherits its settings from the default transport and re-vendor the fixed library. Result: It is now possible to install OCP when using a proxy and custom CA certificates bundle to connect to OpenStack.
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-24 17:14:37 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1925216    
Bug Blocks:    

Comment 1 Martin André 2021-03-26 09:28:52 UTC
I mistakenly made a lite copy of the 4.8 bz instead of a full copy and we lost the comments, so I'm copying below comment https://bugzilla.redhat.com/show_bug.cgi?id=1925216#c10 that summarizes the issue:


Indeed it appears that gophercloud ignores the proxy environment variables when the clouds.yaml includes the cacert property to connect to a cloud using a self-signed certificate.

I've raised the issue in their issue tracker [1] and submitted a fix for it [2]. This code is new in 4.7 which explains why we're not seeing this issue in 4.6.

Once the gophercloud change merges we'll then have to revendor the dependency and backport the change.

[1] https://github.com/gophercloud/utils/issues/148
[2] https://github.com/gophercloud/utils/pull/149

Comment 6 Jon Uriarte 2021-05-20 12:40:59 UTC
Verified in OCP 4.7.0-0.nightly-2021-05-16-105214.

The underlying OSP is 13.0.15 (2021-03-24.1) with TLS (and self-signed cert) in public endpoints enabled.

I've configured a squid proxy so all the requests from the installer host (where I run openshift-install commands from) go through it,
and set the proxy env vars in the installer host:
$ env | grep proxy
https_proxy=https://dummy:dummy@10.46.22.225:3130
http_proxy=http://dummy:dummy@10.46.22.225:3128

The cacert param in the clouds.yaml file in the installer host points to a pem file that includes both proxy's CA cert and
Openstack's CA cert (required as system wide CA trust bundle is not considered anymore).

10.46.22.225 is the proxy's IP (and installer host)
10.46.22.204 is Openstack's API IP

'openshift-install create manifests' command is not ignoring the http_proxy and https_proxy env vars as described in the original bz.


$ ./4.7.0-0.nightly-2021-05-16-105214/openshift-install create install-config --log-level=debug --dir=/home/cloud-user/ostest/                                                                                  
DEBUG OpenShift Installer 4.7.0-0.nightly-2021-05-16-105214                                                                                                                                                                                  
DEBUG Built from commit d585f92b720275809904ba351dddc1d00a50e544
DEBUG Fetching Install Config...
DEBUG Loading Install Config...
DEBUG   Loading SSH Key...
DEBUG   Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Cluster Name...
DEBUG     Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Networking...
DEBUG     Loading Platform...
DEBUG   Loading Pull Secret...
DEBUG   Loading Platform...
DEBUG   Fetching SSH Key...
DEBUG   Generating SSH Key...
DEBUG   Fetching Base Domain...
DEBUG     Fetching Platform...
DEBUG     Generating Platform...
? Platform openstack
? Cloud shiftstack
? ExternalNetwork nova
? APIFloatingIPAddress 10.46.22.244
? FlavorName m4.xlarge
DEBUG   Generating Base Domain...
? Base Domain shiftstack.com
DEBUG   Fetching Cluster Name...
DEBUG     Fetching Base Domain...
DEBUG     Reusing previously-fetched Base Domain
DEBUG     Fetching Platform...
DEBUG     Reusing previously-fetched Platform
DEBUG   Generating Cluster Name...
? Cluster Name newname
DEBUG   Fetching Networking...
DEBUG     Fetching Platform...
DEBUG     Reusing previously-fetched Platform
DEBUG   Generating Networking...
DEBUG   Fetching Pull Secret...
DEBUG   Generating Pull Secret...
? Pull Secret [? for help] ********
DEBUG   Fetching Platform...
DEBUG   Reusing previously-fetched Platform
DEBUG Generating Install Config...
INFO Install-Config created in: /home/cloud-user/ostest

And the proxy's logs show tunnel creation messages (from the installer host to OSP's API):
1621513982.112   4197 10.46.22.225 TCP_TUNNEL/200 4176 CONNECT 10.46.22.204:13774 - HIER_DIRECT/10.46.22.204 -

Comment 8 errata-xmlrpc 2021-05-24 17:14:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.12 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1561