Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1981975

Summary:	Master Machine Config Pool degraded at install time
Product:	OpenShift Container Platform	Reporter:	Daniel Del Ciancio <ddelcian>
Component:	Networking	Assignee:	Luigi Mario Zuccarelli <luzuccar>
Networking sub component:	router	QA Contact:	Hongan Li <hongli>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	aos-bugs, aos-network-edge-staff, cholman, ddelcian, luzuccar, mmasters, raj.sarvaiya
Version:	4.8
Target Milestone:	---
Target Release:	4.9.0
Hardware:	x86_64
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: The config drift seems to happen when the CNO attempts to sanitize the proxy configuration (specifically the no_proxy config). Consequence: It has been observed that a specific IPv6 CIDR missing from the noproxy Fix: Implement logic that updates the dual stack (IPV4 and IPV6) for all scenarios Result: The fix has been verified using verified with 4.9.0-0.nightly-2021-07-25-125326	Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-10-18 17:39:52 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1985588

Comment 4 Miciah Dashiel Butler Masters 2021-07-15 16:15:38 UTC

Based on comment 1, it looks like we need to change this code to account for an install-config with multiple service networks defined, as is the case with dualstack:

	if len(network.Status.ServiceNetwork) > 0 {
		set.Insert(network.Status.ServiceNetwork[0])
	} else {
		return "", fmt.Errorf("serviceNetwork missing from network '%s' status", network.Name)
	}

https://github.com/openshift/cluster-network-operator/blob/18c4ad6453fe4e247d1af6326dfcbdb8ccfdfbca/pkg/util/proxyconfig/no_proxy.go#L77-L81

We'll fix this in 4.9.0 and then evaluate whether we need to backport the fix.

Comment 6 Daniel Del Ciancio 2021-07-16 18:00:09 UTC

Customer has been able to add the IPv6 serviceNetwork CIDR manually to the noproxy configuration and the MCO is no longer in degraded state and the MC update completed successfully on the master nodes.

Could we expect a fix for 4.8 ?  If so, when ? 

Thanks!

Comment 10 Hongan Li 2021-07-26 06:12:13 UTC

verified with 4.9.0-0.nightly-2021-07-25-125326 and the issue has been fixed.

# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-07-25-125326   True        False         19m     Cluster version is 4.9.0-0.nightly-2021-07-25-125326

# oc get network/cluster -oyaml
<---snip---->
status:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  - cidr: fd01::/48
    hostPrefix: 64
  clusterNetworkMTU: 1400
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16
  - fd02::/112

# oc get proxies.config.openshift.io cluster -oyaml
<---snip---->
status:
  httpProxy: http://xxx.redhat.com:xxx
  httpsProxy: http://xxx.redhat.com:xxx
  noProxy: .cluster.local,.svc,10.128.0.0/14,10.73.116.0/23,10.73.a.b,127.0.0.1,172.30.0.0/16,2620:52:0:4974::/64,api-int.bm2-zzhao.qe.devcluster.openshift.com,bm2-zzhao.qe.devcluster.openshift.com,fd01::/48,fd02::/112,localhost


### in old 4.8 version we can see:
status:
  httpProxy: http://xxx.redhat.com:xxx
  httpsProxy: http://xxx.redhat.com:xxx
  noProxy: .cluster.local,.svc,10.128.0.0/14,10.73.116.0/23,10.73.a.b,127.0.0.1,172.30.0.0/16,2620:52:0:4974::/64,api-int.bm2-zzhao.qe.devcluster.openshift.com,bm2-zzhao.qe.devcluster.openshift.com,fd01::/48,localhost

Comment 11 Daniel Del Ciancio 2021-08-17 14:36:19 UTC

Are there plans to backport this to 4.8?

Comment 12 Daniel Del Ciancio 2021-08-18 19:39:47 UTC

My customer tested on 4.8.3 and IPv6 service network CIDR did not appear in the NOPROXY list.  

Do we have an ETA as to when we could expect this fix to land in 4.8.z?

Comment 13 Miciah Dashiel Butler Masters 2021-08-19 21:43:29 UTC

Daniel, please see bug 1985588, which is tracking the 4.8.z backport.  It's currently blocked on CI.  Once it passes CI, it can get cherry-pick approval and merge.  Once a backport merges, it generally will ship a week or two later in the next z-stream release.

Comment 15 Daniel Del Ciancio 2021-09-08 14:08:01 UTC

Customer tested 4.8.10 and proxy configuration looks good.  Tested both fresh cluster install and making the change post cluster install.  Both tests yielded successful results.  Issue can be closed.

Comment 16 Luigi Mario Zuccarelli 2021-09-08 14:48:33 UTC

@ddelcian  - Thanks for the feedback, happy that the fix worked.

Comment 18 errata-xmlrpc 2021-10-18 17:39:52 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759

Comment 19 errata-xmlrpc 2021-10-18 17:52:59 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759