Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1779321

Summary: Failed to verify cert generation after 60 iterations when running etcd-member-recover.sh
Product: OpenShift Container Platform Reporter: Weibin Liang <weliang>
Component: EtcdAssignee: Sam Batschelet <sbatsche>
Status: CLOSED NOTABUG QA Contact: ge liu <geliu>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.3.0CC: mfojtik
Target Milestone: ---Flags: weliang: needinfo-
Target Release: 4.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-12-10 13:36:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Weibin Liang 2019-12-03 18:38:43 UTC
Description of problem:
Follow steps from https://docs.openshift.com/container-platform/4.2/backup_and_restore/disaster_recovery/scenario-1-infra-recovery.html, recoverying failed on step of "Run the etcd-member-recover.sh script."

Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2019-12-02-055401

How reproducible:
Always

Steps to Reproduce:
[root@dhcp-41-193 ~]# ssh -i ~/.ssh/openshift-qe.pem -o StrictHostKeyChecking=no -o ProxyCommand='ssh -A -i ~/.ssh/openshift-qe.pem -o StrictHostKeyChecking=no -o ServerAliveInterval=30 -W %h:%p core.28.30' core.72.247
Warning: Permanently added '10.0.72.247' (ECDSA) to the list of known hosts.
Red Hat Enterprise Linux CoreOS 43.81.201912020253.0
  Part of OpenShift 4.3, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.3/architecture/architecture-rhcos.html

---
[core@ip-10-0-72-247 ~]$ sudo oc login https://localhost:6443
The server uses a certificate signed by an unknown authority.
You can bypass the certificate check, but any data you send to the server could be intercepted by others.
Use insecure connections? (y/n): y

Authentication required for https://localhost:6443 (openshift)
Username: kubeadmin
Password: 
Login successful.

You have access to 53 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".
Welcome! See 'oc help' to get started.
[core@ip-10-0-72-247 ~]$ export SETUP_ETCD_ENVIRONMENT=$(sudo oc adm release info --image-for machine-config-operator --registry-config=/var/lib/kubelet/config.json)
[core@ip-10-0-72-247 ~]$ export KUBE_CLIENT_AGENT=$(sudo oc adm release info --image-for kube-client-agent --registry-config=/var/lib/kubelet/config.json)
[core@ip-10-0-72-247 ~]$ sudo -E /usr/local/bin/etcd-member-recover.sh 10.0.72.247 etcd-member-ip-10-0-57-146.us-east-2.compute.internal
Creating asset directory ./assets
Downloading etcdctl binary..
etcdctl version: 3.3.17
API version: 3.3
Backing up /etc/kubernetes/manifests/etcd-member.yaml to ./assets/backup/
Backing up /etc/etcd/etcd.conf to ./assets/backup/
Trying to backup etcd client certs..
etcd client certs found in /etc/kubernetes/static-pod-resources/kube-apiserver-pod-2 backing up to ./assets/backup/
Stopping etcd..
Waiting for etcd-member to stop
Backing up etcd data-dir..
Backing up etcd certificates..
Removing etcd certs..
Populating template /usr/local/share/openshift-recovery/template/etcd-generate-certs.yaml.template
Populating template ./assets/tmp/etcd-generate-certs.stage1
Populating template ./assets/tmp/etcd-generate-certs.stage2
Starting etcd client cert recovery agent..
Waiting for certs to generate... (1/60)
Waiting for certs to generate... (2/60)
Waiting for certs to generate... (3/60)
Waiting for certs to generate... (4/60)
Waiting for certs to generate... (5/60)
Waiting for certs to generate... (6/60)
Waiting for certs to generate... (7/60)
Waiting for certs to generate... (8/60)
Waiting for certs to generate... (9/60)
Waiting for certs to generate... (10/60)
Waiting for certs to generate... (11/60)
Waiting for certs to generate... (12/60)
Waiting for certs to generate... (13/60)
Waiting for certs to generate... (14/60)
Waiting for certs to generate... (15/60)
Waiting for certs to generate... (16/60)
Waiting for certs to generate... (17/60)
Waiting for certs to generate... (18/60)
Waiting for certs to generate... (19/60)
Waiting for certs to generate... (20/60)
Waiting for certs to generate... (21/60)
Waiting for certs to generate... (22/60)
Waiting for certs to generate... (23/60)
Waiting for certs to generate... (24/60)
Waiting for certs to generate... (25/60)
Waiting for certs to generate... (26/60)
Waiting for certs to generate... (27/60)
Waiting for certs to generate... (28/60)
Waiting for certs to generate... (29/60)
Waiting for certs to generate... (30/60)
Waiting for certs to generate... (31/60)
Waiting for certs to generate... (32/60)
Waiting for certs to generate... (33/60)
Waiting for certs to generate... (34/60)
Waiting for certs to generate... (35/60)
Waiting for certs to generate... (36/60)
Waiting for certs to generate... (37/60)
Waiting for certs to generate... (38/60)
Waiting for certs to generate... (39/60)
Waiting for certs to generate... (40/60)
Waiting for certs to generate... (41/60)
Waiting for certs to generate... (42/60)
Waiting for certs to generate... (43/60)
Waiting for certs to generate... (44/60)
Waiting for certs to generate... (45/60)
Waiting for certs to generate... (46/60)
Waiting for certs to generate... (47/60)
Waiting for certs to generate... (48/60)
Waiting for certs to generate... (49/60)
Waiting for certs to generate... (50/60)
Waiting for certs to generate... (51/60)
Waiting for certs to generate... (52/60)
Waiting for certs to generate... (53/60)
Waiting for certs to generate... (54/60)
Waiting for certs to generate... (55/60)
Waiting for certs to generate... (56/60)
Waiting for certs to generate... (57/60)
Waiting for certs to generate... (58/60)
Waiting for certs to generate... (59/60)
Waiting for certs to generate... (60/60)
Failed to verify cert generation after 60 iterations. Exiting!
[core@ip-10-0-72-247 ~]$ 


Actual results:
Failed to verify cert generation after 60 iterations. Exiting!

Expected results:
Starting etcd..

Additional info:

Comment 1 Sam Batschelet 2019-12-03 19:03:56 UTC
I don't see where you followed 5.a of the docs can you verify?

> Set up a temporary etcd certificate signer service on your master where you have restored etcd.

Comment 2 Weibin Liang 2019-12-03 19:17:51 UTC
(In reply to Sam Batschelet from comment #1)
> I don't see where you followed 5.a of the docs can you verify?
> 
> > Set up a temporary etcd certificate signer service on your master where you have restored etcd.

Here is the log running on the master node (steps 5.a)


[core@ip-10-0-57-146 ~]$ sudo oc login https://localhost:6443
The server uses a certificate signed by an unknown authority.
You can bypass the certificate check, but any data you send to the server could be intercepted by others.
Use insecure connections? (y/n): y

Authentication required for https://localhost:6443 (openshift)
Username: kubeadmin
Password: 
Login successful.

You have access to 53 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".
Welcome! See 'oc help' to get started.
[core@ip-10-0-57-146 ~]$ export KUBE_ETCD_SIGNER_SERVER=$(sudo oc adm release info --image-for kube-etcd-signer-server --registry-config=/var/lib/kubelet/config.json)
[core@ip-10-0-57-146 ~]$ sudo -E /usr/local/bin/tokenize-signer.sh ip-10-0-57-147
Populating template /usr/local/share/openshift-recovery/template/kube-etcd-cert-signer.yaml.template
Populating template ./assets/tmp/kube-etcd-cert-signer.yaml.stage1
Tokenized template now ready: ./assets/manifests/kube-etcd-cert-signer.yaml
[core@ip-10-0-57-146 ~]$ sudo oc create -f assets/manifests/kube-etcd-cert-signer.yaml
pod/etcd-signer created

Comment 3 Sam Batschelet 2019-12-04 20:09:42 UTC
> [core@ip-10-0-57-146 ~]$ sudo -E /usr/local/bin/tokenize-signer.sh ip-10-0-57-147

s/10-0-57-146/ip-10-0-57-147 Typo?

```
section 5.v Verify that the signer is listening on this master node.

[core@ip-10-0-143-125 ~]$ ss -ltn | grep 9943
LISTEN   0         128                       *:9943                   *:*
```

was cert signer running as expected?

Comment 4 Sam Batschelet 2019-12-04 20:10:58 UTC
Sorry added my own typo ..

s/ip-10-0-57-147/ip-10-0-57-146

Comment 5 Weibin Liang 2019-12-10 13:36:38 UTC
Close this bug because I did one step wrong when recovery the failed masters