Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1748263

Summary: [IPI] [OSP] Etcd is not recovered after deleting master instance
Product: OpenShift Container Platform Reporter: David Sanz <dsanzmor>
Component: Etcd OperatorAssignee: Sam Batschelet <sbatsche>
Status: CLOSED DEFERRED QA Contact: ge liu <geliu>
Severity: high Docs Contact:
Priority: high    
Version: 4.2.0CC: agarcial, gblomqui, jhou, mfojtik, sbatsche, scuppett, wjiang
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-02-21 11:58:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1746369    
Bug Blocks:    

Description David Sanz 2019-09-03 08:50:38 UTC
Description of problem:
When deleting master instance, cluster detects it and creates a new one.
API service is restored but there is a container that never ends to boot:

NAMESPACE                                               NAME                                                              READY   STATUS                  RESTARTS   AGE
openshift-etcd                                          etcd-member-morenod-ocp-v9kkx-master-1                            0/2     Init:CrashLoopBackOff   151        21h

ETCD related logs on host:

Sep 03 08:26:28 host-192-168-0-19 hyperkube[1574]: I0903 08:26:28.723539    1574 prober.go:112] Readiness probe for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f):guard" failed (failure):
Sep 03 08:26:28 host-192-168-0-19 hyperkube[1574]: I0903 08:26:28.723786    1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:"spec.containers{guard}"}): type: 'Warning' reason: 'Unhealthy' Readiness probe failed:
Sep 03 08:26:35 host-192-168-0-19 hyperkube[1574]: I0903 08:26:35.375886    1574 file.go:200] Reading config file "/etc/kubernetes/manifests/etcd-member.yaml"
Sep 03 08:26:35 host-192-168-0-19 hyperkube[1574]: I0903 08:26:35.489452    1574 secret.go:186] Setting up volume etcd-client for pod 58c804b7-cdcf-11e9-b79b-fa163e913d1f at /var/lib/kubelet/pods/58c804b7-cdcf-11e9-b79b-fa163e913d1f/volumes/kubernetes.io~secret/etcd-client
Sep 03 08:26:35 host-192-168-0-19 hyperkube[1574]: I0903 08:26:35.489760    1574 configmap.go:191] Setting up volume etcd-serving-ca for pod 58c804b7-cdcf-11e9-b79b-fa163e913d1f at /var/lib/kubelet/pods/58c804b7-cdcf-11e9-b79b-fa163e913d1f/volumes/kubernetes.io~configmap/etcd-serving-ca
Sep 03 08:26:35 host-192-168-0-19 hyperkube[1574]: I0903 08:26:35.492214    1574 secret.go:210] Received secret openshift-apiserver/etcd-client containing (2) pieces of data, 2863 total bytes
Sep 03 08:26:35 host-192-168-0-19 hyperkube[1574]: I0903 08:26:35.492818    1574 configmap.go:215] Received configMap openshift-apiserver/etcd-serving-ca containing (1) pieces of data, 1135 total bytes
Sep 03 08:26:38 host-192-168-0-19 hyperkube[1574]: I0903 08:26:38.729079    1574 prober.go:112] Readiness probe for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f):guard" failed (failure):
Sep 03 08:26:38 host-192-168-0-19 hyperkube[1574]: I0903 08:26:38.729340    1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:"spec.containers{guard}"}): type: 'Warning' reason: 'Unhealthy' Readiness probe failed:
Sep 03 08:26:48 host-192-168-0-19 hyperkube[1574]: I0903 08:26:48.741041    1574 prober.go:112] Readiness probe for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f):guard" failed (failure):
Sep 03 08:26:48 host-192-168-0-19 hyperkube[1574]: I0903 08:26:48.741302    1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:"spec.containers{guard}"}): type: 'Warning' reason: 'Unhealthy' Readiness probe failed:
Sep 03 08:26:55 host-192-168-0-19 hyperkube[1574]: I0903 08:26:55.375729    1574 file.go:200] Reading config file "/etc/kubernetes/manifests/etcd-member.yaml"
Sep 03 08:26:58 host-192-168-0-19 hyperkube[1574]: I0903 08:26:58.985164    1574 prober.go:112] Readiness probe for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f):guard" failed (failure):
Sep 03 08:26:58 host-192-168-0-19 hyperkube[1574]: I0903 08:26:58.985291    1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:"spec.containers{guard}"}): type: 'Warning' reason: 'Unhealthy' Readiness probe failed:
Sep 03 08:27:08 host-192-168-0-19 hyperkube[1574]: I0903 08:27:08.749813    1574 prober.go:112] Readiness probe for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f):guard" failed (failure):
Sep 03 08:27:08 host-192-168-0-19 hyperkube[1574]: I0903 08:27:08.750014    1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:"spec.containers{guard}"}): type: 'Warning' reason: 'Unhealthy' Readiness probe failed:
Sep 03 08:27:15 host-192-168-0-19 hyperkube[1574]: I0903 08:27:15.377196    1574 file.go:200] Reading config file "/etc/kubernetes/manifests/etcd-member.yaml"
Sep 03 08:27:15 host-192-168-0-19 hyperkube[1574]: I0903 08:27:15.425593    1574 kubelet_pods.go:1346] Generating status for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f)"
Sep 03 08:27:15 host-192-168-0-19 hyperkube[1574]: I0903 08:27:15.427121    1574 status_manager.go:382] Ignoring same status for pod "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f)", status: {Phase:Running Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-09-02 15:02:07 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-09-02 15:02:07 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [guard]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-09-02 15:02:07 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [guard]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-09-02 15:02:05 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:192.168.0.19 PodIP:192.168.0.19 StartTime:2019-09-02 15:02:07 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:guard State:{Waiting:nil Running:&ContainerStateRunning{StartedAt:2019-09-02 15:02:14 +0000 UTC,} Terminated:nil} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144fd3d8ead884f3974db906b08fc64ec7535a99fc8a8e2965a34e6a88d072cf ImageID:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144fd3d8ead884f3974db906b08fc64ec7535a99fc8a8e2965a34e6a88d072cf ContainerID:cri-o://1897944f8459c5351b3e652ed053c18ee16323fcac31d030bb469e5f98aab2bb}] QOSClass:Burstable}
Sep 03 08:27:15 host-192-168-0-19 hyperkube[1574]: I0903 08:27:15.427437    1574 volume_manager.go:350] Waiting for volumes to attach and mount for pod "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f)"
Sep 03 08:27:15 host-192-168-0-19 hyperkube[1574]: I0903 08:27:15.427540    1574 volume_manager.go:383] All volumes are attached and mounted for pod "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f)"
Sep 03 08:27:15 host-192-168-0-19 hyperkube[1574]: I0903 08:27:15.430705    1574 kuberuntime_manager.go:617] computePodActions got {KillPod:false CreateSandbox:false SandboxID:d427a9175e127affe2b6273d55011acb2adf11fad2aebb8a8e79c97b9fca897e Attempt:0 NextInitContainerToStart:nil ContainersToStart:[] ContainersToKill:map[]} for pod "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f)"
Sep 03 08:27:15 host-192-168-0-19 hyperkube[1574]: I0903 08:27:15.431271    1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:""}): type: 'Warning' reason: 'DNSConfigForming' Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.0.6 192.168.0.10 192.168.0.12
Sep 03 08:27:18 host-192-168-0-19 hyperkube[1574]: I0903 08:27:18.722708    1574 prober.go:112] Readiness probe for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f):guard" failed (failure):
Sep 03 08:27:18 host-192-168-0-19 hyperkube[1574]: I0903 08:27:18.722869    1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:"spec.containers{guard}"}): type: 'Warning' reason: 'Unhealthy' Readiness probe failed:
Sep 03 08:27:28 host-192-168-0-19 hyperkube[1574]: I0903 08:27:28.726063    1574 prober.go:112] Readiness probe for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f):guard" failed (failure):
Sep 03 08:27:28 host-192-168-0-19 hyperkube[1574]: I0903 08:27:28.726189    1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:"spec.containers{guard}"}): type: 'Warning' reason: 'Unhealthy' Readiness probe failed:
Sep 03 08:27:35 host-192-168-0-19 hyperkube[1574]: I0903 08:27:35.375877    1574 file.go:200] Reading config file "/etc/kubernetes/manifests/etcd-member.yaml"
Sep 03 08:27:38 host-192-168-0-19 hyperkube[1574]: I0903 08:27:38.725832    1574 prober.go:112] Readiness probe for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f):guard" failed (failure):
Sep 03 08:27:38 host-192-168-0-19 hyperkube[1574]: I0903 08:27:38.726182    1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:"spec.containers{guard}"}): type: 'Warning' reason: 'Unhealthy' Readiness probe failed:


==> etcd-member-morenod-ocp-v9kkx-master-1_openshift-etcd_discovery-75d5c14664bcf78c26c70ff17637cdbadcb7da3e670d52c3aab4cfee1f309e82.log <==
2019-09-03T08:39:13.677478215+00:00 stderr F I0903 08:39:13.677337       1 run.go:140] checking against etcd-0.morenod-ocp.qe.devcluster.openshift.com.
2019-09-03T08:39:13.677893420+00:00 stderr F I0903 08:39:13.677859       1 run.go:140] checking against etcd-2.morenod-ocp.qe.devcluster.openshift.com.
2019-09-03T08:39:13.678248085+00:00 stderr F E0903 08:39:13.678222       1 run.go:63] error looking up self: could not find self
2019-09-03T08:39:13.678575547+00:00 stderr F I0903 08:39:13.678552       1 run.go:140] checking against etcd-2.morenod-ocp.qe.devcluster.openshift.com.
2019-09-03T08:39:13.678936010+00:00 stderr F I0903 08:39:13.678913       1 run.go:140] checking against etcd-0.morenod-ocp.qe.devcluster.openshift.com.
2019-09-03T08:39:13.679269015+00:00 stderr F E0903 08:39:13.679244       1 run.go:63] error looking up self: could not find self





Version-Release number of selected component (if applicable):
4.2.0-0.nightly-2019-09-01-224700

How reproducible:


Steps to Reproduce:
1.Install OCP 4.2 on OSP
2.From the OSP console, delete one of the master instances
3.Wait until master instance is created again and check etcd status

Actual results:
etcd is not starting on rebuild node

Expected results:
etcd is recovered

Additional info:

Comment 1 Sam Batschelet 2019-09-03 11:58:53 UTC
This is expected in 4.1.x, 4.2 and will be addressed when cluster-etcd-operator lands in 4.3. For 4.2 we have additional tooling for Disaster Recovery regarding this scenario I will add a link to the docs as soon as it is ready for review.

Parts of this bug including updating DNS and procedures to recover from a failed master node are covered in 4.1 DR documentation[1].

[1] https://docs.openshift.com/container-platform/4.1/disaster_recovery/scenario-1-infra-recovery.html

Comment 2 Greg Blomquist 2019-09-03 16:16:30 UTC
Bumping this out to 4.3.0 ... as Sam pointed out this should be addressed in 4.3 with the cluster-etcd-operator.  And, in the meantime with disaster recovery documentation.

Comment 5 Stephen Cuppett 2020-02-21 11:58:53 UTC
This kind of functionality will be addressed in https://issues.redhat.com/browse/ETCD-55, until then manual DR is required and documented.

Comment 6 Red Hat Bugzilla 2023-09-14 05:42:45 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days