Bug 1943500 - openshift installer fails immediately failed to fetch Install Config
Summary: openshift installer fails immediately failed to fetch Install Config
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.7
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.7.z
Assignee: Martin André
QA Contact: Jon Uriarte
Depends On: 1925216
TreeView+ depends on / blocked
Reported: 2021-03-26 09:20 UTC by Martin André
Modified: 2021-05-24 17:15 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: A change in gophercloud/utils introduced a custom HTTP client to make use of the self-signed certificate specified in the clouds.yaml file. This change however removed all the settings that came with the DefaultTransport, including handling of proxy environment variables and default timeouts. Consequence: Installation that uses both self-signed certificat and proxy fail. Fix: Resolve the issue in gophercloud/utils by ensuring the custom HTTP client inherits its settings from the default transport and re-vendor the fixed library. Result: It is now possible to install OCP when using a proxy and custom CA certificates bundle to connect to OpenStack.
Clone Of:
Last Closed: 2021-05-24 17:14:37 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift installer pull 4794 0 None open Bug 1943500: Bump gophercloud utils 2021-05-04 09:28:53 UTC
Red Hat Product Errata RHSA-2021:1561 0 None None None 2021-05-24 17:15:10 UTC

Comment 1 Martin André 2021-03-26 09:28:52 UTC
I mistakenly made a lite copy of the 4.8 bz instead of a full copy and we lost the comments, so I'm copying below comment https://bugzilla.redhat.com/show_bug.cgi?id=1925216#c10 that summarizes the issue:

Indeed it appears that gophercloud ignores the proxy environment variables when the clouds.yaml includes the cacert property to connect to a cloud using a self-signed certificate.

I've raised the issue in their issue tracker [1] and submitted a fix for it [2]. This code is new in 4.7 which explains why we're not seeing this issue in 4.6.

Once the gophercloud change merges we'll then have to revendor the dependency and backport the change.

[1] https://github.com/gophercloud/utils/issues/148
[2] https://github.com/gophercloud/utils/pull/149

Comment 6 Jon Uriarte 2021-05-20 12:40:59 UTC
Verified in OCP 4.7.0-0.nightly-2021-05-16-105214.

The underlying OSP is 13.0.15 (2021-03-24.1) with TLS (and self-signed cert) in public endpoints enabled.

I've configured a squid proxy so all the requests from the installer host (where I run openshift-install commands from) go through it,
and set the proxy env vars in the installer host:
$ env | grep proxy

The cacert param in the clouds.yaml file in the installer host points to a pem file that includes both proxy's CA cert and
Openstack's CA cert (required as system wide CA trust bundle is not considered anymore). is the proxy's IP (and installer host) is Openstack's API IP

'openshift-install create manifests' command is not ignoring the http_proxy and https_proxy env vars as described in the original bz.

$ ./4.7.0-0.nightly-2021-05-16-105214/openshift-install create install-config --log-level=debug --dir=/home/cloud-user/ostest/                                                                                  
DEBUG OpenShift Installer 4.7.0-0.nightly-2021-05-16-105214                                                                                                                                                                                  
DEBUG Built from commit d585f92b720275809904ba351dddc1d00a50e544
DEBUG Fetching Install Config...
DEBUG Loading Install Config...
DEBUG   Loading SSH Key...
DEBUG   Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Cluster Name...
DEBUG     Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Networking...
DEBUG     Loading Platform...
DEBUG   Loading Pull Secret...
DEBUG   Loading Platform...
DEBUG   Fetching SSH Key...
DEBUG   Generating SSH Key...
DEBUG   Fetching Base Domain...
DEBUG     Fetching Platform...
DEBUG     Generating Platform...
? Platform openstack
? Cloud shiftstack
? ExternalNetwork nova
? APIFloatingIPAddress
? FlavorName m4.xlarge
DEBUG   Generating Base Domain...
? Base Domain shiftstack.com
DEBUG   Fetching Cluster Name...
DEBUG     Fetching Base Domain...
DEBUG     Reusing previously-fetched Base Domain
DEBUG     Fetching Platform...
DEBUG     Reusing previously-fetched Platform
DEBUG   Generating Cluster Name...
? Cluster Name newname
DEBUG   Fetching Networking...
DEBUG     Fetching Platform...
DEBUG     Reusing previously-fetched Platform
DEBUG   Generating Networking...
DEBUG   Fetching Pull Secret...
DEBUG   Generating Pull Secret...
? Pull Secret [? for help] ********
DEBUG   Fetching Platform...
DEBUG   Reusing previously-fetched Platform
DEBUG Generating Install Config...
INFO Install-Config created in: /home/cloud-user/ostest

And the proxy's logs show tunnel creation messages (from the installer host to OSP's API):
1621513982.112   4197 TCP_TUNNEL/200 4176 CONNECT - HIER_DIRECT/ -

Comment 8 errata-xmlrpc 2021-05-24 17:14:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.12 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.