Bug 1748263
| Summary: | [IPI] [OSP] Etcd is not recovered after deleting master instance | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | David Sanz <dsanzmor> |
| Component: | Etcd Operator | Assignee: | Sam Batschelet <sbatsche> |
| Status: | CLOSED DEFERRED | QA Contact: | ge liu <geliu> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 4.2.0 | CC: | agarcial, gblomqui, jhou, mfojtik, sbatsche, scuppett, wjiang |
| Target Milestone: | --- | ||
| Target Release: | 4.4.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-02-21 11:58:53 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1746369 | ||
| Bug Blocks: | |||
This is expected in 4.1.x, 4.2 and will be addressed when cluster-etcd-operator lands in 4.3. For 4.2 we have additional tooling for Disaster Recovery regarding this scenario I will add a link to the docs as soon as it is ready for review. Parts of this bug including updating DNS and procedures to recover from a failed master node are covered in 4.1 DR documentation[1]. [1] https://docs.openshift.com/container-platform/4.1/disaster_recovery/scenario-1-infra-recovery.html Bumping this out to 4.3.0 ... as Sam pointed out this should be addressed in 4.3 with the cluster-etcd-operator. And, in the meantime with disaster recovery documentation. This kind of functionality will be addressed in https://issues.redhat.com/browse/ETCD-55, until then manual DR is required and documented. The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |
Description of problem: When deleting master instance, cluster detects it and creates a new one. API service is restored but there is a container that never ends to boot: NAMESPACE NAME READY STATUS RESTARTS AGE openshift-etcd etcd-member-morenod-ocp-v9kkx-master-1 0/2 Init:CrashLoopBackOff 151 21h ETCD related logs on host: Sep 03 08:26:28 host-192-168-0-19 hyperkube[1574]: I0903 08:26:28.723539 1574 prober.go:112] Readiness probe for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f):guard" failed (failure): Sep 03 08:26:28 host-192-168-0-19 hyperkube[1574]: I0903 08:26:28.723786 1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:"spec.containers{guard}"}): type: 'Warning' reason: 'Unhealthy' Readiness probe failed: Sep 03 08:26:35 host-192-168-0-19 hyperkube[1574]: I0903 08:26:35.375886 1574 file.go:200] Reading config file "/etc/kubernetes/manifests/etcd-member.yaml" Sep 03 08:26:35 host-192-168-0-19 hyperkube[1574]: I0903 08:26:35.489452 1574 secret.go:186] Setting up volume etcd-client for pod 58c804b7-cdcf-11e9-b79b-fa163e913d1f at /var/lib/kubelet/pods/58c804b7-cdcf-11e9-b79b-fa163e913d1f/volumes/kubernetes.io~secret/etcd-client Sep 03 08:26:35 host-192-168-0-19 hyperkube[1574]: I0903 08:26:35.489760 1574 configmap.go:191] Setting up volume etcd-serving-ca for pod 58c804b7-cdcf-11e9-b79b-fa163e913d1f at /var/lib/kubelet/pods/58c804b7-cdcf-11e9-b79b-fa163e913d1f/volumes/kubernetes.io~configmap/etcd-serving-ca Sep 03 08:26:35 host-192-168-0-19 hyperkube[1574]: I0903 08:26:35.492214 1574 secret.go:210] Received secret openshift-apiserver/etcd-client containing (2) pieces of data, 2863 total bytes Sep 03 08:26:35 host-192-168-0-19 hyperkube[1574]: I0903 08:26:35.492818 1574 configmap.go:215] Received configMap openshift-apiserver/etcd-serving-ca containing (1) pieces of data, 1135 total bytes Sep 03 08:26:38 host-192-168-0-19 hyperkube[1574]: I0903 08:26:38.729079 1574 prober.go:112] Readiness probe for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f):guard" failed (failure): Sep 03 08:26:38 host-192-168-0-19 hyperkube[1574]: I0903 08:26:38.729340 1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:"spec.containers{guard}"}): type: 'Warning' reason: 'Unhealthy' Readiness probe failed: Sep 03 08:26:48 host-192-168-0-19 hyperkube[1574]: I0903 08:26:48.741041 1574 prober.go:112] Readiness probe for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f):guard" failed (failure): Sep 03 08:26:48 host-192-168-0-19 hyperkube[1574]: I0903 08:26:48.741302 1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:"spec.containers{guard}"}): type: 'Warning' reason: 'Unhealthy' Readiness probe failed: Sep 03 08:26:55 host-192-168-0-19 hyperkube[1574]: I0903 08:26:55.375729 1574 file.go:200] Reading config file "/etc/kubernetes/manifests/etcd-member.yaml" Sep 03 08:26:58 host-192-168-0-19 hyperkube[1574]: I0903 08:26:58.985164 1574 prober.go:112] Readiness probe for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f):guard" failed (failure): Sep 03 08:26:58 host-192-168-0-19 hyperkube[1574]: I0903 08:26:58.985291 1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:"spec.containers{guard}"}): type: 'Warning' reason: 'Unhealthy' Readiness probe failed: Sep 03 08:27:08 host-192-168-0-19 hyperkube[1574]: I0903 08:27:08.749813 1574 prober.go:112] Readiness probe for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f):guard" failed (failure): Sep 03 08:27:08 host-192-168-0-19 hyperkube[1574]: I0903 08:27:08.750014 1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:"spec.containers{guard}"}): type: 'Warning' reason: 'Unhealthy' Readiness probe failed: Sep 03 08:27:15 host-192-168-0-19 hyperkube[1574]: I0903 08:27:15.377196 1574 file.go:200] Reading config file "/etc/kubernetes/manifests/etcd-member.yaml" Sep 03 08:27:15 host-192-168-0-19 hyperkube[1574]: I0903 08:27:15.425593 1574 kubelet_pods.go:1346] Generating status for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f)" Sep 03 08:27:15 host-192-168-0-19 hyperkube[1574]: I0903 08:27:15.427121 1574 status_manager.go:382] Ignoring same status for pod "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f)", status: {Phase:Running Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-09-02 15:02:07 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-09-02 15:02:07 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [guard]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-09-02 15:02:07 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [guard]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-09-02 15:02:05 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:192.168.0.19 PodIP:192.168.0.19 StartTime:2019-09-02 15:02:07 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:guard State:{Waiting:nil Running:&ContainerStateRunning{StartedAt:2019-09-02 15:02:14 +0000 UTC,} Terminated:nil} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144fd3d8ead884f3974db906b08fc64ec7535a99fc8a8e2965a34e6a88d072cf ImageID:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144fd3d8ead884f3974db906b08fc64ec7535a99fc8a8e2965a34e6a88d072cf ContainerID:cri-o://1897944f8459c5351b3e652ed053c18ee16323fcac31d030bb469e5f98aab2bb}] QOSClass:Burstable} Sep 03 08:27:15 host-192-168-0-19 hyperkube[1574]: I0903 08:27:15.427437 1574 volume_manager.go:350] Waiting for volumes to attach and mount for pod "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f)" Sep 03 08:27:15 host-192-168-0-19 hyperkube[1574]: I0903 08:27:15.427540 1574 volume_manager.go:383] All volumes are attached and mounted for pod "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f)" Sep 03 08:27:15 host-192-168-0-19 hyperkube[1574]: I0903 08:27:15.430705 1574 kuberuntime_manager.go:617] computePodActions got {KillPod:false CreateSandbox:false SandboxID:d427a9175e127affe2b6273d55011acb2adf11fad2aebb8a8e79c97b9fca897e Attempt:0 NextInitContainerToStart:nil ContainersToStart:[] ContainersToKill:map[]} for pod "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f)" Sep 03 08:27:15 host-192-168-0-19 hyperkube[1574]: I0903 08:27:15.431271 1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:""}): type: 'Warning' reason: 'DNSConfigForming' Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 192.168.0.6 192.168.0.10 192.168.0.12 Sep 03 08:27:18 host-192-168-0-19 hyperkube[1574]: I0903 08:27:18.722708 1574 prober.go:112] Readiness probe for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f):guard" failed (failure): Sep 03 08:27:18 host-192-168-0-19 hyperkube[1574]: I0903 08:27:18.722869 1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:"spec.containers{guard}"}): type: 'Warning' reason: 'Unhealthy' Readiness probe failed: Sep 03 08:27:28 host-192-168-0-19 hyperkube[1574]: I0903 08:27:28.726063 1574 prober.go:112] Readiness probe for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f):guard" failed (failure): Sep 03 08:27:28 host-192-168-0-19 hyperkube[1574]: I0903 08:27:28.726189 1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:"spec.containers{guard}"}): type: 'Warning' reason: 'Unhealthy' Readiness probe failed: Sep 03 08:27:35 host-192-168-0-19 hyperkube[1574]: I0903 08:27:35.375877 1574 file.go:200] Reading config file "/etc/kubernetes/manifests/etcd-member.yaml" Sep 03 08:27:38 host-192-168-0-19 hyperkube[1574]: I0903 08:27:38.725832 1574 prober.go:112] Readiness probe for "etcd-quorum-guard-7c7dc46d74-4pclb_openshift-machine-config-operator(716e9aa6-cd91-11e9-9ff5-fa163e913d1f):guard" failed (failure): Sep 03 08:27:38 host-192-168-0-19 hyperkube[1574]: I0903 08:27:38.726182 1574 event.go:209] Event(v1.ObjectReference{Kind:"Pod", Namespace:"openshift-machine-config-operator", Name:"etcd-quorum-guard-7c7dc46d74-4pclb", UID:"716e9aa6-cd91-11e9-9ff5-fa163e913d1f", APIVersion:"v1", ResourceVersion:"113963", FieldPath:"spec.containers{guard}"}): type: 'Warning' reason: 'Unhealthy' Readiness probe failed: ==> etcd-member-morenod-ocp-v9kkx-master-1_openshift-etcd_discovery-75d5c14664bcf78c26c70ff17637cdbadcb7da3e670d52c3aab4cfee1f309e82.log <== 2019-09-03T08:39:13.677478215+00:00 stderr F I0903 08:39:13.677337 1 run.go:140] checking against etcd-0.morenod-ocp.qe.devcluster.openshift.com. 2019-09-03T08:39:13.677893420+00:00 stderr F I0903 08:39:13.677859 1 run.go:140] checking against etcd-2.morenod-ocp.qe.devcluster.openshift.com. 2019-09-03T08:39:13.678248085+00:00 stderr F E0903 08:39:13.678222 1 run.go:63] error looking up self: could not find self 2019-09-03T08:39:13.678575547+00:00 stderr F I0903 08:39:13.678552 1 run.go:140] checking against etcd-2.morenod-ocp.qe.devcluster.openshift.com. 2019-09-03T08:39:13.678936010+00:00 stderr F I0903 08:39:13.678913 1 run.go:140] checking against etcd-0.morenod-ocp.qe.devcluster.openshift.com. 2019-09-03T08:39:13.679269015+00:00 stderr F E0903 08:39:13.679244 1 run.go:63] error looking up self: could not find self Version-Release number of selected component (if applicable): 4.2.0-0.nightly-2019-09-01-224700 How reproducible: Steps to Reproduce: 1.Install OCP 4.2 on OSP 2.From the OSP console, delete one of the master instances 3.Wait until master instance is created again and check etcd status Actual results: etcd is not starting on rebuild node Expected results: etcd is recovered Additional info: