Bug 1925216 - openshift installer fails immediately failed to fetch Install Config
Summary: openshift installer fails immediately failed to fetch Install Config
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.7
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: 4.8.0
Assignee: Martin André
QA Contact: Jon Uriarte
URL:
Whiteboard:
Depends On:
Blocks: 1943500
TreeView+ depends on / blocked
 
Reported: 2021-02-04 15:57 UTC by Ilan Green
Modified: 2021-07-27 22:42 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: A change in gophercloud/utils introduced a custom HTTP client to make use of the self-signed certificate specified in the clouds.yaml file. This change however removed all the settings that came with the DefaultTransport, including handling of proxy environment variables and default timeouts. Consequence: Installation that uses both self-signed certificat and proxy fail. Fix: Resolve the issue in gophercloud/utils by ensuring the custom HTTP client inherits its settings from the default transport and re-vendor the fixed library. Result: It is now possible to install OCP when using a proxy and custom CA certificates bundle to connect to OpenStack.
Clone Of:
Environment:
Last Closed: 2021-07-27 22:41:34 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 4786 0 None open Bug 1925216: Bump gophercloud utils 2021-03-24 07:36:27 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:42:14 UTC

Comment 4 Martin André 2021-02-10 16:43:37 UTC
Ilan, can this be closed per your comment https://bugzilla.redhat.com/show_bug.cgi?id=1917587#c15 ?

Comment 10 Martin André 2021-03-17 13:42:35 UTC
Indeed it appears that gophercloud ignores the proxy environment variables when the clouds.yaml includes the cacert property to connect to a cloud using a self-signed certificate.

I've raised the issue in their issue tracker [1] and submitted a fix for it [2]. This code is new in 4.7 which explains why we're not seeing this issue in 4.6.

Once the gophercloud change merges we'll then have to revendor the dependency and backport the change.

[1] https://github.com/gophercloud/utils/issues/148
[2] https://github.com/gophercloud/utils/pull/149

Comment 11 Ilan Green 2021-03-17 15:01:24 UTC
(In reply to Martin André from comment #10)
> Indeed it appears that gophercloud ignores the proxy environment variables
> when the clouds.yaml includes the cacert property to connect to a cloud
> using a self-signed certificate.
> 
> I've raised the issue in their issue tracker [1] and submitted a fix for it
> [2]. This code is new in 4.7 which explains why we're not seeing this issue
> in 4.6.
> 
> Once the gophercloud change merges we'll then have to revendor the
> dependency and backport the change.
> 
> [1] https://github.com/gophercloud/utils/issues/148
> [2] https://github.com/gophercloud/utils/pull/149

Thanks for looking into it
Is there any option to workaround it in the meantime?

Comment 12 Martin André 2021-03-17 15:39:37 UTC
(In reply to Ilan Green from comment #11)
> (In reply to Martin André from comment #10)
> > Indeed it appears that gophercloud ignores the proxy environment variables
> > when the clouds.yaml includes the cacert property to connect to a cloud
> > using a self-signed certificate.
> > 
> > I've raised the issue in their issue tracker [1] and submitted a fix for it
> > [2]. This code is new in 4.7 which explains why we're not seeing this issue
> > in 4.6.
> > 
> > Once the gophercloud change merges we'll then have to revendor the
> > dependency and backport the change.
> > 
> > [1] https://github.com/gophercloud/utils/issues/148
> > [2] https://github.com/gophercloud/utils/pull/149
> 
> Thanks for looking into it
> Is there any option to workaround it in the meantime?

Unfortunately, no. The library we use to talk to openstack has a bug and there is no way to work around it.

Comment 15 Jon Uriarte 2021-04-21 12:38:43 UTC
Hello,

I'm verifying this BZ in OCP 4.8.0-0.nightly-2021-04-18-101412 as it's working as expected, although I was not able to reproduce the
original issue in my environment.

These are my findings so far when verifying this bz.

The underlying OSP is 13.0.16 (2021-04-09.1) with TLS (and self-signed cert) in public endpoints enabled, so it's required the CA cert in
the server that makes the queries to OSP API.
I have tried with different OCP versions: 4.6.16, 4.7.0-fc.5 and latest 4.8, to compare results and trying to reproduce the original issue.

I've configured a squid proxy so all the requests from the installer host (where I run openshift-install commands from) go through it,
and set the proxy env vars in the installer host:
$ env | grep proxy
https_proxy=https://dummy:dummy@10.46.22.239:3130
http_proxy=http://dummy:dummy@10.46.22.239:3128

The proxy has Openstack's CA cert in it's trusted bundle, and the installer host has the proxy's CA cert in it's trusted bundle.

10.46.22.239 is the proxy's IP
10.46.22.204 is Openstack's IP

As I understand the issue described in this BZ is that 'openshift-install create manifests' command is ignoring the http_proxy and
https_proxy env vars.

What I see from my tests:

------
4.6.16
------

$ ./4.6.16/openshift-install create install-config --log-level=debug --dir=/home/cloud-user/ostest/                                                                                                            
DEBUG OpenShift Installer 4.6.16
DEBUG Built from commit 8a1ec01353e68cb6ebb1dd890d684f885c33145a
DEBUG Fetching Install Config...
DEBUG Loading Install Config...
DEBUG   Loading SSH Key...
DEBUG   Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Cluster Name...
DEBUG     Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Pull Secret...
DEBUG   Loading Platform...
DEBUG   Fetching SSH Key...
DEBUG   Generating SSH Key...
DEBUG   Fetching Base Domain...
DEBUG     Fetching Platform...
DEBUG     Generating Platform...
? Platform openstack
? Cloud shiftstack
? ExternalNetwork nova
? APIFloatingIPAddress 10.46.22.225
? FlavorName m1.medium
DEBUG   Generating Base Domain...
? Base Domain shiftstack.com
DEBUG   Fetching Cluster Name...
DEBUG     Fetching Base Domain...
DEBUG     Reusing previously-fetched Base Domain
DEBUG     Fetching Platform...
DEBUG     Reusing previously-fetched Platform
DEBUG   Generating Cluster Name...
? Cluster Name bla
DEBUG   Fetching Pull Secret...
DEBUG   Generating Pull Secret...
? Pull Secret [? for help] **
DEBUG   Fetching Platform...
DEBUG   Reusing previously-fetched Platform
DEBUG Generating Install Config...


The connection to Openstack API is stablished through the proxy:
1618999698.083  10641 10.46.22.239 TCP_TUNNEL/200 9755 CONNECT 10.46.22.204:13000 - HIER_DIRECT/10.46.22.204 - (from /var/log/squid/access.log)

It looks like the the cacert param in the clouds.yaml is ignored, and the trusted CA bundle from the system is used.

----------
4.7.0-fc.5
----------

$ ./4.7.0-fc.5/openshift-install create install-config --log-level=debug --dir=/home/cloud-user/ostest/                                                                                                        
DEBUG OpenShift Installer 4.7.0-fc.5                                                                                                                                                                                                         
DEBUG Built from commit 8e0274dbba909dfb4fce42865c213ec117138eeb
DEBUG Fetching Install Config...
DEBUG Loading Install Config...
DEBUG   Loading SSH Key...
DEBUG   Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Cluster Name...
DEBUG     Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Networking...
DEBUG     Loading Platform...
DEBUG   Loading Pull Secret...
DEBUG   Loading Platform...
DEBUG   Fetching SSH Key...
DEBUG   Generating SSH Key...
DEBUG   Fetching Base Domain...
DEBUG     Fetching Platform...
DEBUG     Generating Platform...
? Platform openstack
? Cloud shiftstack
? ExternalNetwork nova
? APIFloatingIPAddress 10.46.22.225
? FlavorName m4.xlarge
DEBUG   Generating Base Domain...
? Base Domain shitstack.com
DEBUG   Fetching Cluster Name...
DEBUG     Fetching Base Domain...
DEBUG     Reusing previously-fetched Base Domain
DEBUG     Fetching Platform...
DEBUG     Reusing previously-fetched Platform
DEBUG   Generating Cluster Name...
? Cluster Name bla
DEBUG   Fetching Networking...
DEBUG     Fetching Platform...
DEBUG     Reusing previously-fetched Platform
DEBUG   Generating Networking...
DEBUG   Fetching Pull Secret...
DEBUG   Generating Pull Secret...
? Pull Secret [? for help] **
DEBUG   Fetching Platform...
DEBUG   Reusing previously-fetched Platform
DEBUG Generating Install Config...
INFO Install-Config created in: /home/cloud-user/ostest

The connection to Openstack API is stablished through the proxy:
1619000144.170  20137 10.46.22.239 TCP_TUNNEL/200 26355 CONNECT 10.46.22.204:13000 - HIER_DIRECT/10.46.22.204 - (from /var/log/squid/access.log)

So what I see is that the proxy env vars are not being ignored.

As in 4.6 , it looks like the the cacert param in the clouds.yaml is ignored, and the trusted CA bundle from the system is used.

---------------------------------
4.8.0-0.nightly-2021-04-18-101412
---------------------------------


$ ./4.8.0-0.nightly-2021-04-18-101412/openshift-install create install-config --log-level=debug --dir=/home/cloud-user/ostest/                                                                                 
DEBUG OpenShift Installer 4.8.0-0.nightly-2021-04-18-101412                                                                                                                                                                                  
DEBUG Built from commit 907ba997eebc2a5795763d8496e36df7d1fdc51f                                                                                                                                                                             
DEBUG Fetching Install Config...                                                                                                                                                                                                             
DEBUG Loading Install Config...
DEBUG   Loading SSH Key...
DEBUG   Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Cluster Name...
DEBUG     Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Networking...
DEBUG     Loading Platform...
DEBUG   Loading Pull Secret...
DEBUG   Loading Platform...
DEBUG   Fetching SSH Key...
DEBUG   Generating SSH Key...
DEBUG   Fetching Base Domain...
DEBUG     Fetching Platform...
DEBUG     Generating Platform...
? Platform openstack
? Cloud shiftstack
? ExternalNetwork nova
? APIFloatingIPAddress 10.46.22.225
? FlavorName m4.xlarge
DEBUG   Generating Base Domain...
? Base Domain shitstack.com
DEBUG   Fetching Cluster Name...
DEBUG     Fetching Base Domain...
DEBUG     Reusing previously-fetched Base Domain
DEBUG     Fetching Platform...
DEBUG     Reusing previously-fetched Platform
DEBUG   Generating Cluster Name...
? Cluster Name bla
DEBUG   Fetching Networking...
DEBUG     Fetching Platform...
DEBUG     Reusing previously-fetched Platform
DEBUG   Generating Networking...
DEBUG   Fetching Pull Secret...
DEBUG   Generating Pull Secret...
? Pull Secret [? for help] **
DEBUG   Fetching Platform...
DEBUG   Reusing previously-fetched Platform
DEBUG Generating Install Config...
INFO Install-Config created in: /home/cloud-user/ostest

Here I found a difference with 4.6 and 4.7, as the cacert param from the clouds.yaml is considered instead of
system wide CA trust bundle.

cacert param needs to point to a pem file that includes both proxy's CA cert and Openstack's CA cert (this is
the difference with 4.6 and 4.7, where the cacert param wasn't considered in favor of system wide CA trust
bundle).

About when to land in 4.7, it will be backported once it's verified in 4.8.

Comment 19 errata-xmlrpc 2021-07-27 22:41:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.