Bug 1949150

Summary: Bootstrap ignition shim doesn't follow proxy settings
Product: OpenShift Container Platform Reporter: Emilien Macchi <emacchi>
Component: InstallerAssignee: Emilien Macchi <emacchi>
Installer sub component: OpenShift on OpenStack QA Contact: Jon Uriarte <juriarte>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: juriarte, m.andre, openshift-bugzilla-robot, pprinett
Version: 4.7Keywords: Triaged
Target Milestone: ---   
Target Release: 4.6.z   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1947427 Environment:
Last Closed: 2021-07-28 05:51:20 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1947427    
Bug Blocks:    

Comment 5 Jon Uriarte 2021-07-12 20:45:40 UTC
Verified in 4.6.0-0.nightly-2021-07-07-181104 on top of OSP 13.0.15 (2021-03-24.1) with TLS (and self-signed cert)
in public endpoints enabled.

A squid proxy has been deployed and configured in install-config.yaml:
proxy:
    httpProxy: "http://dummy:dummy@10.46.22.225:3128"
    httpsProxy: "https://dummy:dummy@10.46.22.225:3130"
    noProxy: oauth-openshift.apps.ostest.shiftstack.com

In OCP 4.6 the proxy's CA cert must be in the system wide CA bundle, as the cacert param from the clouds.yaml is not being considered.
On the other hand, the proxy has Openstack's CA cert in it's trusted bundle.

10.46.22.225 is the proxy's IP
10.46.22.204:13292 is Openstack Glance API endpoint
10.46.22.230 is the bootstrap VM's IP

$ openstack server list                                                                                                                                                                            
+--------------------------------------+------------------------+--------+----------------------------------------------------+---------------------------------------+-----------+
| ID                                   | Name                   | Status | Networks                                           | Image                                 | Flavor    |
+--------------------------------------+------------------------+--------+----------------------------------------------------+---------------------------------------+-----------+
|...                                                        
| 60e3c0ce-e11b-4b5b-8087-5f0f91a737ae | ostest-9d7cw-bootstrap | ACTIVE | ostest-9d7cw-openshift=10.196.2.10, 10.46.22.230   | ostest-9d7cw-rhcos                    | m4.xlarge |
|...
+--------------------------------------+------------------------+--------+----------------------------------------------------+---------------------------------------+-----------+

The bootstrap VM shows this message when booting up (note the host it queries is the final glance endpoint):
[    9.774640] ignition[714]: GET https://10.46.22.204:13292//v2/images/8a294493-2c41-4388-bccb-a1d6344733f2/file: attempt #1
[   11.777226] ignition[714]: GET result: OK

And the access.log squid file shows:
1626120972.042   2024 10.46.22.243 TCP_TUNNEL/200 323132 CONNECT 10.46.22.204:13292 - HIER_DIRECT/10.46.22.204 -

I've run additional comprobations by stopping the proxy server when the bootstrap VM is booting up, and got this messages from the VM:

[    9.534681] ignition[707]: GET https://10.46.22.204:13292//v2/images/f81548ec-b0a2-4f47-8146-51c21008f17a/file: attempt #1
[    9.540315] ignition[707]: GET error: Get "https://10.46.22.204:13292//v2/images/f81548ec-b0a2-4f47-8146-51c21008f17a/file": proxyconnect tcp: dial tcp 10.46.22.225:3130: connect: connection refused
[    9.734921] ignition[707]: GET https://10.46.22.204:13292//v2/images/f81548ec-b0a2-4f47-8146-51c21008f17a/file: attempt #2
[    9.740904] ignition[707]: GET error: Get "https://10.46.22.204:13292//v2/images/f81548ec-b0a2-4f47-8146-51c21008f17a/file": proxyconnect tcp: dial tcp 10.46.22.225:3130: connect: connection refused
[   10.141348] ignition[707]: GET https://10.46.22.204:13292//v2/images/f81548ec-b0a2-4f47-8146-51c21008f17a/file: attempt #3
[   10.148258] ignition[707]: GET error: Get "https://10.46.22.204:13292//v2/images/f81548ec-b0a2-4f47-8146-51c21008f17a/file": proxyconnect tcp: dial tcp 10.46.22.225:3130: connect: connection refused
[   10.948550] ignition[707]: GET https://10.46.22.204:13292//v2/images/f81548ec-b0a2-4f47-8146-51c21008f17a/file: attempt #4
[   10.955662] ignition[707]: GET error: Get "https://10.46.22.204:13292//v2/images/f81548ec-b0a2-4f47-8146-51c21008f17a/file": proxyconnect tcp: dial tcp 10.46.22.225:3130: connect: connection refused
[   12.556324] ignition[707]: GET https://10.46.22.204:13292//v2/images/f81548ec-b0a2-4f47-8146-51c21008f17a/file: attempt #5
[   12.563702] ignition[707]: GET error: Get "https://10.46.22.204:13292//v2/images/f81548ec-b0a2-4f47-8146-51c21008f17a/file": proxyconnect tcp: dial tcp 10.46.22.225:3130: connect: connection refused

which means
 1. the GET request is towards OSP glance API (10.46.22.204), so it's transparent proxy
 2. the proxy is correctly configured as it fails with the connection refused from the proxy (10.46.22.225)

Comment 8 errata-xmlrpc 2021-07-28 05:51:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.40 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2767