Bug 2047501

Summary:	etcd static pod does not come up
Product:	OpenShift Container Platform	Reporter:	Dan Williams <dcbw>
Component:	Node	Assignee:	Harshal Patil <harpatil>
Node sub component:	Kubelet	QA Contact:	Sunil Choudhary <schoudha>
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	unspecified
Priority:	unspecified	CC:	aos-bugs, harpatil, rphillips
Version:	4.10
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-01-28 14:48:39 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Dan Williams 2022-01-27 22:30:50 UTC

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_ovn-kubernetes/934/pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn/1486787220873416704

Operator degraded (ClusterMemberController_Error::DefragController_Error::EtcdMembers_UnhealthyMembers::StaticPods_Error): ClusterMemberControllerDegraded: unhealthy members found during reconciling members
DefragControllerDegraded: cluster is unhealthy: 2 of 3 members are available, ip-10-0-175-62.ec2.internal is unhealthy
EtcdMembersDegraded: 2 of 3 members are available, ip-10-0-175-62.ec2.internal is unhealthy
StaticPodsDegraded: pod/etcd-ip-10-0-135-32.ec2.internal container "etcd-health-monitor" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=etcd-health-monitor pod=etcd-ip-10-0-135-32.ec2.internal_openshift-etcd(a4fe506f8dad1b11823a403d5e2a5ea9)
StaticPodsDegraded: pod/etcd-ip-10-0-175-62.ec2.internal container "etcd" started at 2022-01-27 20:13:01 +0000 UTC is still not ready

It looks like Kubelet sees the static pod written by installer-6 (openshift-etcd/etcd-ip-10-0-175-62.ec2.internal) but it never actually gets started.

Jan 27 20:13:36.481057 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.481027    2001 kubelet.go:2096] "SyncLoop REMOVE" source="file" pods=[openshift-etcd/etcd-ip-10-0-175-62.ec2.internal]
Jan 27 20:13:36.482291 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.482275    2001 kubelet.go:2086] "SyncLoop ADD" source="file" pods=[openshift-etcd/etcd-ip-10-0-175-62.ec2.internal]
...
Jan 27 20:13:36.624597 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.624575    2001 status_manager.go:614] "Pod was deleted and then recreated, skipping status update" pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal" oldPodUID=ea6fa363d16d85891fbc2fd15c694840 podUID=bc8572ebe332f47bd8fb3b6fb99150eb
...
Jan 27 20:13:36.632473 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.632449    2001 status_manager.go:614] "Pod was deleted and then recreated, skipping status update" pod="openshift-etcd/etcd-ip-10-0-175-62.ec
2.internal" oldPodUID=ea6fa363d16d85891fbc2fd15c694840 podUID=bc8572ebe332f47bd8fb3b6fb99150eb
...
Jan 27 20:13:37.089442 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:37.088989    2001 kuberuntime_container.go:720] "Killing container with a grace period" pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal" po
dUID=ea6fa363d16d85891fbc2fd15c694840 containerName="etcdctl" containerID="cri-o://9d7fdae6c973f83ed1780a21821c99e476cd4017af42598c5d12c791453e2a80" gracePeriod=1
Jan 27 20:13:37.089442 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:37.089012    2001 kuberuntime_container.go:720] "Killing container with a grace period" pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal" po
dUID=ea6fa363d16d85891fbc2fd15c694840 containerName="etcd" containerID="cri-o://712348ea74eda7c11ae7bdccef296be3882ce8c3b9ca8324d68a7f0dbb782f16" gracePeriod=1
Jan 27 20:13:37.090954 ip-10-0-175-62 crio[1891]: time="2022-01-27 20:13:37.090930361Z" level=info msg="Stopped container 9d7fdae6c973f83ed1780a21821c99e476cd4017af42598c5d12c791453e2a80: openshift-etcd/etcd-ip-
10-0-175-62.ec2.internal/etcdctl" id=cb2a6dc3-9207-4981-999f-b09203f2c0f0 name=/runtime.v1.RuntimeService/StopContainer
Jan 27 20:13:37.091538 ip-10-0-175-62 crio[1891]: time="2022-01-27 20:13:37.091518027Z" level=info msg="Stopped container 712348ea74eda7c11ae7bdccef296be3882ce8c3b9ca8324d68a7f0dbb782f16: openshift-etcd/etcd-ip-
10-0-175-62.ec2.internal/etcd" id=ab021d64-be2c-4b49-b634-4b600e54627d name=/runtime.v1.RuntimeService/StopContainer

We'd expect pod bc8572ebe332f47bd8fb3b6fb99150eb to be started, I think?

The last installer instance (installer-6) writes out the static pod yaml at 20:13:36. From when kubelet notices the change:

$ grep ip-10-0-175-62.ec2.internal /tmp/journal | grep etcd | grep -v etcd_installer | grep -v installer-6
Jan 27 20:13:36.481057 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.481027    2001 kubelet.go:2096] "SyncLoop REMOVE" source="file" pods=[openshift-etcd/etcd-ip-10-0-175-62.ec2.internal]
Jan 27 20:13:36.482291 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.482275    2001 kubelet.go:2086] "SyncLoop ADD" source="file" pods=[openshift-etcd/etcd-ip-10-0-175-62.ec2.internal]
Jan 27 20:13:36.529666 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.529639    2001 reconciler.go:216] "operationExecutor.VerifyControllerAttachedVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-resource-dir\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.529666 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.529688    2001 reconciler.go:216] "operationExecutor.VerifyControllerAttachedVolume started for volume \"static-pod-dir\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-static-pod-dir\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.529878 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.529712    2001 reconciler.go:216] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-dir\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-data-dir\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.529878 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.529816    2001 reconciler.go:216] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-cert-dir\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.529878 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.529844    2001 reconciler.go:216] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-dir\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-log-dir\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.529878 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.529877    2001 reconciler.go:216] "operationExecutor.VerifyControllerAttachedVolume started for volume \"usr-local-bin\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-usr-local-bin\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.617798 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.617774    2001 logs.go:319] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-ip-10-0-175-62.ec2.internal_ea6fa363d16d85891fbc2fd15c694840/etcd-health-monitor/0.log"
Jan 27 20:13:36.618248 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.618235    2001 logs.go:319] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-ip-10-0-175-62.ec2.internal_ea6fa363d16d85891fbc2fd15c694840/etcd-metrics/0.log"
Jan 27 20:13:36.618622 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.618609    2001 logs.go:319] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-ip-10-0-175-62.ec2.internal_ea6fa363d16d85891fbc2fd15c694840/etcd/0.log"
Jan 27 20:13:36.618980 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.618968    2001 logs.go:319] "Finished parsing log file" path="/var/log/pods/openshift-etcd_etcd-ip-10-0-175-62.ec2.internal_ea6fa363d16d85891fbc2fd15c694840/etcdctl/0.log"
Jan 27 20:13:36.624597 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.624575    2001 status_manager.go:614] "Pod was deleted and then recreated, skipping status update" pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal" oldPodUID=ea6fa363d16d85891fbc2fd15c694840 podUID=bc8572ebe332f47bd8fb3b6fb99150eb
Jan 27 20:13:36.631183 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.631162    2001 reconciler.go:253] "operationExecutor.MountVolume started for volume \"data-dir\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-data-dir\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.631254 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.631235    2001 reconciler.go:253] "operationExecutor.MountVolume started for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-cert-dir\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.631284 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.631259    2001 reconciler.go:253] "operationExecutor.MountVolume started for volume \"log-dir\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-log-dir\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.631310 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.631284    2001 reconciler.go:253] "operationExecutor.MountVolume started for volume \"usr-local-bin\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-usr-local-bin\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.631310 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.631307    2001 reconciler.go:253] "operationExecutor.MountVolume started for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-resource-dir\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.631396 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.631342    2001 reconciler.go:253] "operationExecutor.MountVolume started for volume \"static-pod-dir\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-static-pod-dir\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.631396 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.631396    2001 operation_generator.go:755] "MountVolume.SetUp succeeded for volume \"static-pod-dir\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-static-pod-dir\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.631437 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.631433    2001 operation_generator.go:755] "MountVolume.SetUp succeeded for volume \"data-dir\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-data-dir\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.631701 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.631660    2001 operation_generator.go:755] "MountVolume.SetUp succeeded for volume \"cert-dir\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-cert-dir\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.631753 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.631742    2001 operation_generator.go:755] "MountVolume.SetUp succeeded for volume \"log-dir\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-log-dir\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.631796 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.631787    2001 operation_generator.go:755] "MountVolume.SetUp succeeded for volume \"usr-local-bin\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-usr-local-bin\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.631835 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.631818    2001 operation_generator.go:755] "MountVolume.SetUp succeeded for volume \"resource-dir\" (UniqueName: \"kubernetes.io/host-path/bc8572ebe332f47bd8fb3b6fb99150eb-resource-dir\") pod \"etcd-ip-10-0-175-62.ec2.internal\" (UID: \"bc8572ebe332f47bd8fb3b6fb99150eb\") " pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal"
Jan 27 20:13:36.632473 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:36.632449    2001 status_manager.go:614] "Pod was deleted and then recreated, skipping status update" pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal" oldPodUID=ea6fa363d16d85891fbc2fd15c694840 podUID=bc8572ebe332f47bd8fb3b6fb99150eb
Jan 27 20:13:37.089442 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:37.088989    2001 kuberuntime_container.go:720] "Killing container with a grace period" pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal" podUID=ea6fa363d16d85891fbc2fd15c694840 containerName="etcdctl" containerID="cri-o://9d7fdae6c973f83ed1780a21821c99e476cd4017af42598c5d12c791453e2a80" gracePeriod=1
Jan 27 20:13:37.089442 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:37.089012    2001 kuberuntime_container.go:720] "Killing container with a grace period" pod="openshift-etcd/etcd-ip-10-0-175-62.ec2.internal" podUID=ea6fa363d16d85891fbc2fd15c694840 containerName="etcd" containerID="cri-o://712348ea74eda7c11ae7bdccef296be3882ce8c3b9ca8324d68a7f0dbb782f16" gracePeriod=1
Jan 27 20:13:37.090954 ip-10-0-175-62 crio[1891]: time="2022-01-27 20:13:37.090930361Z" level=info msg="Stopped container 9d7fdae6c973f83ed1780a21821c99e476cd4017af42598c5d12c791453e2a80: openshift-etcd/etcd-ip-10-0-175-62.ec2.internal/etcdctl" id=cb2a6dc3-9207-4981-999f-b09203f2c0f0 name=/runtime.v1.RuntimeService/StopContainer
Jan 27 20:13:37.091538 ip-10-0-175-62 crio[1891]: time="2022-01-27 20:13:37.091518027Z" level=info msg="Stopped container 712348ea74eda7c11ae7bdccef296be3882ce8c3b9ca8324d68a7f0dbb782f16: openshift-etcd/etcd-ip-10-0-175-62.ec2.internal/etcd" id=ab021d64-be2c-4b49-b634-4b600e54627d name=/runtime.v1.RuntimeService/StopContainer
Jan 27 20:13:45.503052 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:45.503057    2001 reconciler.go:295] "Volume detached for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/2db9d696-bb56-4b11-a703-b662861d1859-etcd-serving-ca\") on node \"ip-10-0-175-62.ec2.internal\" DevicePath \"\""
Jan 27 20:13:45.503241 ip-10-0-175-62 hyperkube[2001]: I0127 20:13:45.503113    2001 reconciler.go:295] "Volume detached for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/2db9d696-bb56-4b11-a703-b662861d1859-etcd-client\") on node \"ip-10-0-175-62.ec2.internal\" DevicePath \"\""
Jan 27 20:17:07.814704 ip-10-0-175-62 hyperkube[2001]: I0127 20:17:07.814531    2001 reconciler.go:295] "Volume detached for volume \"etcd-serving-ca\" (UniqueName: \"kubernetes.io/configmap/6f8d44ac-bd83-436f-92fa-281c920d11d4-etcd-serving-ca\") on node \"ip-10-0-175-62.ec2.internal\" DevicePath \"\""
Jan 27 20:17:07.814704 ip-10-0-175-62 hyperkube[2001]: I0127 20:17:07.814550    2001 reconciler.go:295] "Volume detached for volume \"etcd-client\" (UniqueName: \"kubernetes.io/secret/6f8d44ac-bd83-436f-92fa-281c920d11d4-etcd-client\") on node \"ip-10-0-175-62.ec2.internal\" DevicePath \"\""
Jan 27 20:19:01.302786 ip-10-0-175-62 crio[1891]: time="2022-01-27 20:19:01.302751269Z" level=info msg="Removed container 5a445c40f939d19f9a78b986385614eef2043d6264cafbed9ce7b1a95fef0c37: openshift-etcd/etcd-ip-10-0-175-62.ec2.internal/etcd-ensure-env-vars" id=fe3d05cf-71bb-4497-b287-34a6dc47560d name=/runtime.v1.RuntimeService/RemoveContainer
Jan 27 20:19:01.332538 ip-10-0-175-62 crio[1891]: time="2022-01-27 20:19:01.332496839Z" level=info msg="Removed container a76a93f8133c9e26bc7064be141cb397451f230347cb8c803a7ca212c6435f29: openshift-etcd/etcd-ip-10-0-175-62.ec2.internal/setup" id=de2fe109-af08-4504-8f0b-b8110c585f1e name=/runtime.v1.RuntimeService/RemoveContainer
Jan 27 20:19:01.388972 ip-10-0-175-62 crio[1891]: time="2022-01-27 20:19:01.388934290Z" level=info msg="Removed container 13d0f6835c841047cd5f8b05ed1d72f17e72b5b130570645e85cb0fb54969169: openshift-etcd/etcd-ip-10-0-175-62.ec2.internal/etcd-resources-copy" id=ebfe9f2a-bfa3-4899-8cb9-0d0c2eb0e905 name=/runtime.v1.RuntimeService/RemoveContainer
Jan 27 20:20:01.484827 ip-10-0-175-62 crio[1891]: time="2022-01-27 20:20:01.484790057Z" level=info msg="Removed container e5fdee51872ecf6e704c7301dca4c6a095cbe84fd1eec6b7085ce0945a3b2f6d: openshift-etcd/etcd-ip-10-0-175-62.ec2.internal/etcd-metrics" id=3638d3c6-ed41-4e2c-8df0-08e9a8aa423c name=/runtime.v1.RuntimeService/RemoveContainer
Jan 27 20:20:01.587443 ip-10-0-175-62 crio[1891]: time="2022-01-27 20:20:01.587414126Z" level=info msg="Removed container 669c5a17f0c18fb03c0336d559ba61510f16c260fe534d2570575e989f9b8a5f: openshift-etcd/etcd-ip-10-0-175-62.ec2.internal/etcd-health-monitor" id=4f6f7db2-beab-4a9b-b1bd-37c9c357b113 name=/runtime.v1.RuntimeService/RemoveContainer
Jan 27 20:20:01.601976 ip-10-0-175-62 crio[1891]: time="2022-01-27 20:20:01.601947637Z" level=info msg="Removed container 9d7fdae6c973f83ed1780a21821c99e476cd4017af42598c5d12c791453e2a80: openshift-etcd/etcd-ip-10-0-175-62.ec2.internal/etcdctl" id=6c01b8c7-ec9a-4e65-90a6-c57d5cad7e14 name=/runtime.v1.RuntimeService/RemoveContainer
Jan 27 20:20:01.655537 ip-10-0-175-62 crio[1891]: time="2022-01-27 20:20:01.655496486Z" level=info msg="Removed container 712348ea74eda7c11ae7bdccef296be3882ce8c3b9ca8324d68a7f0dbb782f16: openshift-etcd/etcd-ip-10-0-175-62.ec2.internal/etcd" id=a2a27cd8-ddaf-4f38-be0a-444fff2d9676 name=/runtime.v1.RuntimeService/RemoveContainer

Comment 1 Ryan Phillips 2022-01-28 14:48:39 UTC


*** This bug has been marked as a duplicate of bug 2040533 ***