Bug 1897546 - Backup taken on one master cannot be restored on other masters
Summary: Backup taken on one master cannot be restored on other masters
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Etcd
Version: 4.5
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.4.z
Assignee: Suresh Kolichala
QA Contact: ge liu
Depends On: 1897543
TreeView+ depends on / blocked
Reported: 2020-11-13 11:58 UTC by Suresh Kolichala
Modified: 2021-02-03 10:12 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1897543
Last Closed: 2021-02-03 10:11:43 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift cluster-etcd-operator pull 506 0 None closed Bug 1897546: DR: Make restore pod yaml specific to restoring node 2021-02-01 18:29:29 UTC
Red Hat Product Errata RHSA-2021:0281 0 None None None 2021-02-03 10:12:28 UTC

Description Suresh Kolichala 2020-11-13 11:58:40 UTC
+++ This bug was initially created as a clone of Bug #1897543 +++

+++ This bug was initially created as a clone of Bug #1897542 +++

+++ This bug was initially created as a clone of Bug #1895509 +++

Description of problem:
A recent change in 4.5 introduced a bug that disallows backups taken on one master to be restored on other masters.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Take a cluster-backup using cluster-backup.sh on one master.
2. Copy the backup on to another master.
3. Attempt to restore the database on the other master using procedure documented.

Actual results:
The etcds fail to come up on all masters.

Expected results:
All etcds successfully start on all masters and the cluster recovers.

Additional info:

--- Additional comment from Suresh Kolichala on 2020-11-06 20:40:14 UTC ---

As a workaround, a backup should be restored on the same master it is taken from. 

To determine the master where the backup is taken from, one may run the following command against the backup directory:

sudo tar xvzf <backup>/static_kuberesources_*.tar.gz  *restore-etcd-pod/pod.yaml --to-stdout  2>&1 | grep 'ETCD_NAME='

Comment 3 ge liu 2020-12-23 08:31:58 UTC
Tried with 4.4.0-0.ci.test-2020-12-23-042527-ci-ln-cs764w2

got backup from master-0, then restore with this backup on master-2, it succeed.
# oc get pods -n openshift-etcd | grep etcd
etcd-ip-10-0-139-137.us-east-2.compute.internal                4/4     Running     0          3m34s
etcd-ip-10-0-168-222.us-east-2.compute.internal                4/4     Running     0          2m33s
etcd-ip-10-0-205-133.us-east-2.compute.internal                4/4     Running     0          3m3s

Comment 5 ge liu 2021-01-21 09:35:42 UTC
Verified at comments 3, but I just curious about why bug status have not updated to 'verified' since I have executed pre-merge verification.

Comment 8 errata-xmlrpc 2021-02-03 10:11:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.4.33 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.