Bug 1791993

Summary: [IPI baremetal] proxy-based install fails due to mismatch in no_proxy configuration in installer vs. cluster-network-operator
Product: OpenShift Container Platform Reporter: Stephen Benjamin <stbenjam>
Component: InstallerAssignee: Stephen Benjamin <stbenjam>
Installer sub component: OpenShift on Bare Metal IPI QA Contact: Victor Voronkov <vvoronko>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: unspecified CC: augol, rbartal
Version: 4.4   
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1791995 (view as bug list) Environment:
Last Closed: 2020-05-04 11:24:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1791995    

Description Stephen Benjamin 2020-01-16 20:15:50 UTC 
Description of problem:

The installer creates a manifest for proxy configuration, automatically
adding specific addresses to NO_PROXY depending on the platform. One of
those addresses is the metadata service, hosted at 169.254.169.254. The
installer assumes this must be done for all platforms other than None of
vSphere, whereas the cluster-network-operator has an explicit list of
platforms:

https://github.com/openshift/cluster-network-operator/blob/adaf257b4d63661726443ab2b059a9b4209a02d1/pkg/util/proxyconfig/no_proxy.go#L67-L69

When using a proxy with baremetal IPI, the installer adds this address,
however when the CNO comes up, it does not, causing the rendered
machine configs to differ, and installation to fail, with MCO reporting
errors like:

pool master has not progressed to latest configuration: configuration
status for pool master is empty: pool is degraded because nodes fail
with "3 nodes are reporting degraded status on sync": "Node master-1 is
reporting: \"machineconfig.machineconfiguration.openshift.io
\\\"rendered-master-982b8698753da7e31b5f902aa4dc135e\\\" not found\""



Version-Release number of the following components:

Seen in:
4.3.0-0.nightly-2020-01-11-070223
4.4.0-0.nightly-2020-01-12-221811


How reproducible:

Always

Steps to Reproduce:
1. Install baremetal IPI with a proxy enabled

Actual results:

Installation failure with these errors:

pool master has not progressed to latest configuration: configuration
status for pool master is empty: pool is degraded because nodes fail
with "3 nodes are reporting degraded status on sync": "Node master-1 is
reporting: \"machineconfig.machineconfiguration.openshift.io
\\\"rendered-master-982b8698753da7e31b5f902aa4dc135e\\\" not found\""


Expected results:

Install succeeds

Additional info:
Comment 2 Victor Voronkov 2020-02-20 10:34:35 UTC 
Verified in Virtual IPv4 environment on build 4.4.0-0.nightly-2020-02-19-044512

with squid as a proxy w/o authentication as container on hypervisor

using jenkins job https://jenkins-fci-continuous-productization.cloud.paas.psi.redhat.com/job/vvoronko-test-pipeline/

[kni@provisionhost-0 ~]$ cat install-config.yaml
apiVersion: v1
baseDomain: qe.lab.redhat.com
proxy:
  httpProxy: http://192.168.123.1:3128
  httpsProxy: http://192.168.123.1:3128
  noProxy: 172.22.0.0/24
networking:
  networkType: OpenShiftSDN
  machineCIDR: 192.168.123.0/24
metadata:
  name: vvoronko-cluster
...

Deployment finished successfully
Comment 4 errata-xmlrpc 2020-05-04 11:24:47 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581