Bug 1882176

Summary:	Master recovery instructions don't handle IP change well
Product:	OpenShift Container Platform	Reporter:	Jim Minter <jminter>
Component:	Etcd	Assignee:	Suresh Kolichala <skolicha>
Status:	CLOSED ERRATA	QA Contact:	ge liu <geliu>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.4	CC:	agarcial, cdepaula, dramseur, ffranz, mjudeiki, sbatsche, wking
Target Milestone:	---
Target Release:	4.7.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: If IP address is changed on a master node, the certs on the disk are no longer valid but etcd error messages do not report the real problem. Consequence: The end user is not clear as to why etcd is failing to connect with peers. Fix: Detect the change in IP address, and report the event and mark EtcdCertSignerController as Degraded. Result:	Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-02-24 15:19:20 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jim Minter 2020-09-24 01:18:52 UTC

In 4.4, ARO had to recreate a failed master VM.  In the process of recreation, the master IP changed.  The instructions at [0] are inadequate to recover full etcd membership in this case and it was necessary to manually delete the etcd-peer- Secret for master-0 to cause a new certificate to be created with the new IP address.

There are a couple of asks here:

1- The above use case needs to be added to documentation and QE flows.
2- Ideally, it would be good if the cert signer were able to automatically detect changes of IP and do them.  Failing that, better documentation is needed.

[0] https://docs.openshift.com/container-platform/4.4/backup_and_restore/replacing-unhealthy-etcd-member.html#restore-replace-crashlooping-etcd-member_replacing-unhealthy-etcd-member

Comment 1 Sam Batschelet 2020-09-24 11:51:27 UTC

> 1- The above use case needs to be added to documentation and QE flows.

As the below backport might take a bit to land in 4.4 we will address docs as p[0].

> 2- Ideally, it would be good if the cert signer were able to automatically detect changes of IP and do them.

I 100% agree that we can check the SAN[1] and invalidate certs based on IP SAN not matching hostIP for the node.

This is not blocking 4.6 but will be addressed with high priority, thanks for the report.

[1] https://github.com/openshift/cluster-etcd-operator/blob/release-4.4/pkg/operator/etcdcertsigner/etcdcertsignercontroller.go#L282

Comment 3 Suresh Kolichala 2020-10-09 14:29:45 UTC

*** Bug 1886771 has been marked as a duplicate of this bug. ***

Comment 8 errata-xmlrpc 2021-02-24 15:19:20 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633