Bug 1882176 - Master recovery instructions don't handle IP change well
Summary: Master recovery instructions don't handle IP change well
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Etcd
Version: 4.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.7.0
Assignee: Suresh Kolichala
QA Contact: ge liu
URL:
Whiteboard:
: 1886771 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-24 01:18 UTC by Jim Minter
Modified: 2021-02-24 15:20 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: If IP address is changed on a master node, the certs on the disk are no longer valid but etcd error messages do not report the real problem. Consequence: The end user is not clear as to why etcd is failing to connect with peers. Fix: Detect the change in IP address, and report the event and mark EtcdCertSignerController as Degraded. Result:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:19:20 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-etcd-operator pull 461 0 None closed Bug 1882176: Validate certs for the current IP address of the node 2021-02-18 04:23:44 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:20:58 UTC

Description Jim Minter 2020-09-24 01:18:52 UTC
In 4.4, ARO had to recreate a failed master VM.  In the process of recreation, the master IP changed.  The instructions at [0] are inadequate to recover full etcd membership in this case and it was necessary to manually delete the etcd-peer- Secret for master-0 to cause a new certificate to be created with the new IP address.

There are a couple of asks here:

1- The above use case needs to be added to documentation and QE flows.
2- Ideally, it would be good if the cert signer were able to automatically detect changes of IP and do them.  Failing that, better documentation is needed.

[0] https://docs.openshift.com/container-platform/4.4/backup_and_restore/replacing-unhealthy-etcd-member.html#restore-replace-crashlooping-etcd-member_replacing-unhealthy-etcd-member

Comment 1 Sam Batschelet 2020-09-24 11:51:27 UTC
> 1- The above use case needs to be added to documentation and QE flows.

As the below backport might take a bit to land in 4.4 we will address docs as p[0].

> 2- Ideally, it would be good if the cert signer were able to automatically detect changes of IP and do them.

I 100% agree that we can check the SAN[1] and invalidate certs based on IP SAN not matching hostIP for the node.

This is not blocking 4.6 but will be addressed with high priority, thanks for the report.

[1] https://github.com/openshift/cluster-etcd-operator/blob/release-4.4/pkg/operator/etcdcertsigner/etcdcertsignercontroller.go#L282

Comment 3 Suresh Kolichala 2020-10-09 14:29:45 UTC
*** Bug 1886771 has been marked as a duplicate of this bug. ***

Comment 8 errata-xmlrpc 2021-02-24 15:19:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.