Bug 1755560 - Performing disaster recovery twice on a cluster can fail to proceed the second time
Summary: Performing disaster recovery twice on a cluster can fail to proceed the secon...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Etcd
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.3.0
Assignee: Clayton Coleman
QA Contact: ge liu
URL:
Whiteboard:
Depends On:
Blocks: 1755557
TreeView+ depends on / blocked
 
Reported: 2019-09-25 16:55 UTC by Clayton Coleman
Modified: 2020-05-13 21:25 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1755557
Environment:
Last Closed: 2020-05-13 21:25:19 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 1133 0 'None' closed Bug 1755560: Ensure ETCD_INITIAL_CLUSTER is preserved during restore 2020-04-17 09:12:12 UTC
Red Hat Product Errata RHBA-2020:0062 0 None None None 2020-05-13 21:25:21 UTC

Description Clayton Coleman 2019-09-25 16:55:29 UTC
+++ This bug was initially created as a clone of Bug #1755557 +++

Scenario:

1. Perform DR steps (remove two masters, wait until quorum lost, physically delete machines)
2. Recover following instructions
3. Repeat 1 but pick a new master (one that was created in step 2)

Expected:

Able to start a new etcd quorum on the first master with only itself

Actual:

/usr/local/bin/etcd-snapshot-restore.sh /root/assets/backup/snapshot.db etcd-member-ip-10-0-149-142.ec2.internal=https://etcd-2.ci-ln-shp2psk-d5d6b.origin-ci-int-aws.dev.rhcloud.com:2380

restored the cluster containing the previous (etcd-0, etcd-1) members, preventing further progress.


The root cause is that the etcd-snapshot-restore script accepts ETCD_INITIAL_CLUSTER as an argument, but then sources /run/etcd/environment which may contain the value ETCD_INITIAL_CLUSTER.  So the user's intent was to start a new cluster with one member, but we started with 3 members (including the two that are permanently gone).

The script needs to preserve ETCD_INITIAL_CLUSTER correctly even if it's set in /run/etcd/environment.

Needs to be back ported to all releases.

Comment 3 Clayton Coleman 2019-10-16 17:42:22 UTC
This is high severity, I need this verified soon so we can get into 4.1 and 4.2 (and enable the disruptive suites to prevent future regressions).

Comment 4 ge liu 2019-10-29 11:14:32 UTC
Verified with 4.3.

Comment 7 errata-xmlrpc 2020-05-13 21:25:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062


Note You need to log in before you can comment on or make changes to this bug.