Bug 1945236

Summary:	Bootstrap ignition shim doesn't follow proxy settings
Product:	OpenShift Container Platform	Reporter:	Emilien Macchi <emacchi>
Component:	Installer	Assignee:	Emilien Macchi <emacchi>
Installer sub component:	OpenShift on OpenStack	QA Contact:	Jon Uriarte <juriarte>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	medium	Keywords:	Triaged
Version:	4.8
Target Milestone:	---
Target Release:	4.8.0
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-07-27 22:56:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1947427

Description Emilien Macchi 2021-03-31 13:53:45 UTC

Version:

$ openshift-install version
4.8 (current master

Platform: OpenStack / IPI

What happened?

When doing a disconnected install, using HTTP proxy to reach external networks, the bootstrap ignition shim isn't aware of the Proxy env, and tries to reach the OpenStack Glance server without proxy; which is problematic in disconnected environment.

What did you expect to happen?

The traffic to reach Glance should go through the proxy.

How to reproduce it (as minimally and precisely as possible)?

Run the deployment with:
proxy:
  httpProxy: http://your-proxy

Bootstrap will not reach ignition file via the proxy.

Comment 2 Jon Uriarte 2021-05-12 12:39:54 UTC

Verified in 4.8.0-0.nightly-2021-05-07-011724 on top of OSP 13.0.15 (2021-03-24.1) with TLS (and self-signed cert)
in public endpoints enabled.

A squid proxy has been deployed and configured in install-config.yaml:
proxy:
    httpProxy: "http://dummy:dummy@10.46.22.225:3128"
    httpsProxy: "https://dummy:dummy@10.46.22.225:3130"
    noProxy: oauth-openshift.apps.ostest.shiftstack.com

In OCP 4.8, as the cacert param from the clouds.yaml is considered instead of system wide CA trust bundle, the cacert
param in clouds.yaml needs to point to a pem file that includes both proxy's CA cert and Openstack's CA cert.

On the other hand, the proxy has Openstack's CA cert in it's trusted bundle.

10.46.22.225 is the proxy's IP
10.46.22.204:13292 is Openstack Glance API endpoint
10.46.22.227 is the bootstrap VM's IP

$ openstack server list                                                                                                                                                                            
+--------------------------------------+------------------------+--------+----------------------------------------------------+---------------------------------------+-----------+                                                          
| ID                                   | Name                   | Status | Networks                                           | Image                                 | Flavor    |                                                          
+--------------------------------------+------------------------+--------+----------------------------------------------------+---------------------------------------+-----------+                                                          
|...                                                        
| 1c105c2c-715e-4eb9-a273-b62e50fb8a88 | ostest-jkdd4-bootstrap | ACTIVE | ostest-jkdd4-openshift=10.196.3.155, 10.46.22.227  | ostest-jkdd4-rhcos                    | m4.xlarge |                                                          
|...
+--------------------------------------+------------------------+--------+----------------------------------------------------+---------------------------------------+-----------+    

The bootstrap VM shows this message when booting up:
[   10.238539] ignition[710]: GET https://10.46.22.204:13292//v2/images/3f617377-5b30-4a6f-a8dc-ce7cdefce4fe/file: attempt #1
[   12.489092] ignition[710]: GET result: OK

And the access.log squid file shows:
1620815018.229   1303 10.46.22.227 TCP_TUNNEL/200 290642 CONNECT 10.46.22.204:13292 - HIER_DIRECT/10.46.22.204 -

which means the tunnel between the bootstrap (10.46.22.227) and OSP glance endpoint (10.46.22.204:13292) has been stablished.

I've run additional comprobations by stopping the proxy server when the bootstrap VM is booting up, and got this messages from the VM:

[   10.167655] ignition[708]: GET https://10.46.22.204:13292//v2/images/0cbba449-71ba-49ec-a86b-b6cbf4e7c40f/file: attempt #1
[   10.253845] ignition[708]: GET error: Get "https://10.46.22.204:13292//v2/images/0cbba449-71ba-49ec-a86b-b6cbf4e7c40f/file": proxyconnect tcp: dial tcp 10.46.22.225:3130: connect: connection refused
[   10.454077] ignition[708]: GET https://10.46.22.204:13292//v2/images/0cbba449-71ba-49ec-a86b-b6cbf4e7c40f/file: attempt #2
[   10.460508] ignition[708]: GET error: Get "https://10.46.22.204:13292//v2/images/0cbba449-71ba-49ec-a86b-b6cbf4e7c40f/file": proxyconnect tcp: dial tcp 10.46.22.225:3130: connect: connection refused
[   10.856208] ignition[708]: GET https://10.46.22.204:13292//v2/images/0cbba449-71ba-49ec-a86b-b6cbf4e7c40f/file: attempt #3
[   10.862020] ignition[708]: GET error: Get "https://10.46.22.204:13292//v2/images/0cbba449-71ba-49ec-a86b-b6cbf4e7c40f/file": proxyconnect tcp: dial tcp 10.46.22.225:3130: connect: connection refused
[   11.658310] ignition[708]: GET https://10.46.22.204:13292//v2/images/0cbba449-71ba-49ec-a86b-b6cbf4e7c40f/file: attempt #4
[   11.663929] ignition[708]: GET error: Get "https://10.46.22.204:13292//v2/images/0cbba449-71ba-49ec-a86b-b6cbf4e7c40f/file": proxyconnect tcp: dial tcp 10.46.22.225:3130: connect: connection refused
[   13.260938] ignition[708]: GET https://10.46.22.204:13292//v2/images/0cbba449-71ba-49ec-a86b-b6cbf4e7c40f/file: attempt #5
[   13.267148] ignition[708]: GET error: Get "https://10.46.22.204:13292//v2/images/0cbba449-71ba-49ec-a86b-b6cbf4e7c40f/file": proxyconnect tcp: dial tcp 10.46.22.225:3130: connect: connection refused

which means
 1. the GET request is towards OSP glance API (10.46.22.204), so it's transparent proxy
 2. the proxy is correctly configured as it fails with the connection refused from the proxy (10.46.22.225)

Comment 5 errata-xmlrpc 2021-07-27 22:56:48 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438