Bug 1925216
| Summary: | openshift installer fails immediately failed to fetch Install Config | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Ilan Green <igreen> |
| Component: | Installer | Assignee: | Martin André <m.andre> |
| Installer sub component: | OpenShift on OpenStack | QA Contact: | Jon Uriarte <juriarte> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | eduen, m.andre, mfedosin, nstielau, pprinett |
| Version: | 4.7 | Keywords: | Triaged |
| Target Milestone: | --- | ||
| Target Release: | 4.8.0 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: A change in gophercloud/utils introduced a custom HTTP client to make use of the self-signed certificate specified in the clouds.yaml file. This change however removed all the settings that came with the DefaultTransport, including handling of proxy environment variables and default timeouts.
Consequence: Installation that uses both self-signed certificat and proxy fail.
Fix: Resolve the issue in gophercloud/utils by ensuring the custom HTTP client inherits its settings from the default transport and re-vendor the fixed library.
Result: It is now possible to install OCP when using a proxy and custom CA certificates bundle to connect to OpenStack.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-07-27 22:41:34 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1943500 | ||
|
Comment 4
Martin André
2021-02-10 16:43:37 UTC
Indeed it appears that gophercloud ignores the proxy environment variables when the clouds.yaml includes the cacert property to connect to a cloud using a self-signed certificate. I've raised the issue in their issue tracker [1] and submitted a fix for it [2]. This code is new in 4.7 which explains why we're not seeing this issue in 4.6. Once the gophercloud change merges we'll then have to revendor the dependency and backport the change. [1] https://github.com/gophercloud/utils/issues/148 [2] https://github.com/gophercloud/utils/pull/149 (In reply to Martin André from comment #10) > Indeed it appears that gophercloud ignores the proxy environment variables > when the clouds.yaml includes the cacert property to connect to a cloud > using a self-signed certificate. > > I've raised the issue in their issue tracker [1] and submitted a fix for it > [2]. This code is new in 4.7 which explains why we're not seeing this issue > in 4.6. > > Once the gophercloud change merges we'll then have to revendor the > dependency and backport the change. > > [1] https://github.com/gophercloud/utils/issues/148 > [2] https://github.com/gophercloud/utils/pull/149 Thanks for looking into it Is there any option to workaround it in the meantime? (In reply to Ilan Green from comment #11) > (In reply to Martin André from comment #10) > > Indeed it appears that gophercloud ignores the proxy environment variables > > when the clouds.yaml includes the cacert property to connect to a cloud > > using a self-signed certificate. > > > > I've raised the issue in their issue tracker [1] and submitted a fix for it > > [2]. This code is new in 4.7 which explains why we're not seeing this issue > > in 4.6. > > > > Once the gophercloud change merges we'll then have to revendor the > > dependency and backport the change. > > > > [1] https://github.com/gophercloud/utils/issues/148 > > [2] https://github.com/gophercloud/utils/pull/149 > > Thanks for looking into it > Is there any option to workaround it in the meantime? Unfortunately, no. The library we use to talk to openstack has a bug and there is no way to work around it. Hello, I'm verifying this BZ in OCP 4.8.0-0.nightly-2021-04-18-101412 as it's working as expected, although I was not able to reproduce the original issue in my environment. These are my findings so far when verifying this bz. The underlying OSP is 13.0.16 (2021-04-09.1) with TLS (and self-signed cert) in public endpoints enabled, so it's required the CA cert in the server that makes the queries to OSP API. I have tried with different OCP versions: 4.6.16, 4.7.0-fc.5 and latest 4.8, to compare results and trying to reproduce the original issue. I've configured a squid proxy so all the requests from the installer host (where I run openshift-install commands from) go through it, and set the proxy env vars in the installer host: $ env | grep proxy https_proxy=https://dummy:dummy@10.46.22.239:3130 http_proxy=http://dummy:dummy@10.46.22.239:3128 The proxy has Openstack's CA cert in it's trusted bundle, and the installer host has the proxy's CA cert in it's trusted bundle. 10.46.22.239 is the proxy's IP 10.46.22.204 is Openstack's IP As I understand the issue described in this BZ is that 'openshift-install create manifests' command is ignoring the http_proxy and https_proxy env vars. What I see from my tests: ------ 4.6.16 ------ $ ./4.6.16/openshift-install create install-config --log-level=debug --dir=/home/cloud-user/ostest/ DEBUG OpenShift Installer 4.6.16 DEBUG Built from commit 8a1ec01353e68cb6ebb1dd890d684f885c33145a DEBUG Fetching Install Config... DEBUG Loading Install Config... DEBUG Loading SSH Key... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Cluster Name... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Pull Secret... DEBUG Loading Platform... DEBUG Fetching SSH Key... DEBUG Generating SSH Key... DEBUG Fetching Base Domain... DEBUG Fetching Platform... DEBUG Generating Platform... ? Platform openstack ? Cloud shiftstack ? ExternalNetwork nova ? APIFloatingIPAddress 10.46.22.225 ? FlavorName m1.medium DEBUG Generating Base Domain... ? Base Domain shiftstack.com DEBUG Fetching Cluster Name... DEBUG Fetching Base Domain... DEBUG Reusing previously-fetched Base Domain DEBUG Fetching Platform... DEBUG Reusing previously-fetched Platform DEBUG Generating Cluster Name... ? Cluster Name bla DEBUG Fetching Pull Secret... DEBUG Generating Pull Secret... ? Pull Secret [? for help] ** DEBUG Fetching Platform... DEBUG Reusing previously-fetched Platform DEBUG Generating Install Config... The connection to Openstack API is stablished through the proxy: 1618999698.083 10641 10.46.22.239 TCP_TUNNEL/200 9755 CONNECT 10.46.22.204:13000 - HIER_DIRECT/10.46.22.204 - (from /var/log/squid/access.log) It looks like the the cacert param in the clouds.yaml is ignored, and the trusted CA bundle from the system is used. ---------- 4.7.0-fc.5 ---------- $ ./4.7.0-fc.5/openshift-install create install-config --log-level=debug --dir=/home/cloud-user/ostest/ DEBUG OpenShift Installer 4.7.0-fc.5 DEBUG Built from commit 8e0274dbba909dfb4fce42865c213ec117138eeb DEBUG Fetching Install Config... DEBUG Loading Install Config... DEBUG Loading SSH Key... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Cluster Name... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Networking... DEBUG Loading Platform... DEBUG Loading Pull Secret... DEBUG Loading Platform... DEBUG Fetching SSH Key... DEBUG Generating SSH Key... DEBUG Fetching Base Domain... DEBUG Fetching Platform... DEBUG Generating Platform... ? Platform openstack ? Cloud shiftstack ? ExternalNetwork nova ? APIFloatingIPAddress 10.46.22.225 ? FlavorName m4.xlarge DEBUG Generating Base Domain... ? Base Domain shitstack.com DEBUG Fetching Cluster Name... DEBUG Fetching Base Domain... DEBUG Reusing previously-fetched Base Domain DEBUG Fetching Platform... DEBUG Reusing previously-fetched Platform DEBUG Generating Cluster Name... ? Cluster Name bla DEBUG Fetching Networking... DEBUG Fetching Platform... DEBUG Reusing previously-fetched Platform DEBUG Generating Networking... DEBUG Fetching Pull Secret... DEBUG Generating Pull Secret... ? Pull Secret [? for help] ** DEBUG Fetching Platform... DEBUG Reusing previously-fetched Platform DEBUG Generating Install Config... INFO Install-Config created in: /home/cloud-user/ostest The connection to Openstack API is stablished through the proxy: 1619000144.170 20137 10.46.22.239 TCP_TUNNEL/200 26355 CONNECT 10.46.22.204:13000 - HIER_DIRECT/10.46.22.204 - (from /var/log/squid/access.log) So what I see is that the proxy env vars are not being ignored. As in 4.6 , it looks like the the cacert param in the clouds.yaml is ignored, and the trusted CA bundle from the system is used. --------------------------------- 4.8.0-0.nightly-2021-04-18-101412 --------------------------------- $ ./4.8.0-0.nightly-2021-04-18-101412/openshift-install create install-config --log-level=debug --dir=/home/cloud-user/ostest/ DEBUG OpenShift Installer 4.8.0-0.nightly-2021-04-18-101412 DEBUG Built from commit 907ba997eebc2a5795763d8496e36df7d1fdc51f DEBUG Fetching Install Config... DEBUG Loading Install Config... DEBUG Loading SSH Key... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Cluster Name... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Networking... DEBUG Loading Platform... DEBUG Loading Pull Secret... DEBUG Loading Platform... DEBUG Fetching SSH Key... DEBUG Generating SSH Key... DEBUG Fetching Base Domain... DEBUG Fetching Platform... DEBUG Generating Platform... ? Platform openstack ? Cloud shiftstack ? ExternalNetwork nova ? APIFloatingIPAddress 10.46.22.225 ? FlavorName m4.xlarge DEBUG Generating Base Domain... ? Base Domain shitstack.com DEBUG Fetching Cluster Name... DEBUG Fetching Base Domain... DEBUG Reusing previously-fetched Base Domain DEBUG Fetching Platform... DEBUG Reusing previously-fetched Platform DEBUG Generating Cluster Name... ? Cluster Name bla DEBUG Fetching Networking... DEBUG Fetching Platform... DEBUG Reusing previously-fetched Platform DEBUG Generating Networking... DEBUG Fetching Pull Secret... DEBUG Generating Pull Secret... ? Pull Secret [? for help] ** DEBUG Fetching Platform... DEBUG Reusing previously-fetched Platform DEBUG Generating Install Config... INFO Install-Config created in: /home/cloud-user/ostest Here I found a difference with 4.6 and 4.7, as the cacert param from the clouds.yaml is considered instead of system wide CA trust bundle. cacert param needs to point to a pem file that includes both proxy's CA cert and Openstack's CA cert (this is the difference with 4.6 and 4.7, where the cacert param wasn't considered in favor of system wide CA trust bundle). About when to land in 4.7, it will be backported once it's verified in 4.8. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438 |