Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1960772

Summary: InstallerPodContainerWaiting_ContainerCreating unable to fetch pod log stats
Product: OpenShift Container Platform Reporter: W. Trevor King <wking>
Component: NodeAssignee: Elana Hashman <ehashman>
Node sub component: Kubelet QA Contact: Sunil Choudhary <schoudha>
Status: CLOSED DUPLICATE Docs Contact:
Severity: medium    
Priority: medium CC: aos-bugs
Version: 4.8   
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-02 22:35:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description W. Trevor King 2021-05-14 19:53:47 UTC 
Similar to bug 1959495, but with different underlying kubelet error messages.  In 4.8.0-fc.3 -> 4.8.0-fc.4 update CI [1]:


  operator conditions kube-scheduler	0s
  Operator degraded (InstallerPodContainerWaiting_ContainerCreating): 
    InstallerPodContainerWaitingDegraded: Pod "installer-6-ci-ln-9ciwnj2-002ac-mdjkv-master-1" on node "ci-ln-9ciwnj2-002ac-mdjkv-master-1" container "installer" is waiting since 2021-05-14 15:50:33 +0000 UTC because ContainerCreating

  [sig-arch][Early] Managed cluster should start all core operators [Skipped:Disconnected] [Suite:openshift/conformance/parallel]	1s
  fail [github.com/onsi/ginkgo.0- 
 origin.0+incompatible/internal/leafnodes/runner.go:113]: May 14 16:08:37.493: 
    Some cluster operators are not ready: kube-scheduler (Degraded=True InstallerPodContainerWaiting_ContainerCreating: InstallerPodContainerWaitingDegraded: Pod "installer-6-ci-ln-9ciwnj2-002ac-mdjkv-master-1" on node "ci-ln-9ciwnj2-002ac-mdjkv-master-1" container "installer" is waiting since 2021-05-14 15:50:33 +0000 UTC because ContainerCreating)

  [sig-instrumentation] Prometheus when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Early] [Skipped:Disconnected] [Suite:openshift/conformance/parallel]	1m10s
  [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial] 
  disruption_tests: [sig-arch] Check if alerts are firing during or after upgrade success	2h32m31s
  May 14 18:42:18.123: Unexpected alerts fired or pending during the upgrade:

    alert ClusterOperatorDegraded fired for 157 seconds with labels: {condition="Degraded", endpoint="metrics", instance="10.0.0.6:9099", job="cluster-version-operator", name="kube-scheduler", namespace="openshift-cluster-version", pod="cluster-version-operator-6bc96f7897-hlvnc", reason="InstallerPodContainerWaiting_ContainerCreating", service="cluster-version-operator", severity="critical"}
    alert ClusterOperatorDegraded fired for 7188 seconds with labels: {condition="Degraded", endpoint="metrics", instance="10.0.0.5:9099", job="cluster-version-operator", name="kube-scheduler", namespace="openshift-cluster-version", pod="cluster-version-operator-75654f68f7-w5fdq", reason="InstallerPodContainerWaiting_ContainerCreating", service="cluster-version-operator", severity="warning"}
    alert KubeContainerWaiting fired for 6331 seconds with labels: {container="installer", namespace="openshift-kube-scheduler", pod="installer-6-ci-ln-9ciwnj2-002ac-mdjkv-master-1", severity="warning"}
alert KubePodNotReady fired for 9031 seconds with labels: {namespace="openshift-kube-scheduler", pod="installer-6-ci-ln-9ciwnj2-002ac-mdjkv-master-1", severity="warning"}

 $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/release-openshift-origin-installer-launch-azure/1393223397898457088/artifacts/launch/nodes/ci-ln-9ciwnj2-002ac-mdjkv-master-1/journal | gunzip | grep -1 installer-6-ci-ln-9ciwnj2-002ac-mdjkv-master-1 | tail -n3
May 14 18:43:20.320812 ci-ln-9ciwnj2-002ac-mdjkv-master-1 hyperkube[1589]: I0514 18:43:20.320784    1589 pod_container_manager_linux.go:198] "Failed to delete cgroup paths" cgroupName=[kubepods pod9c34afa1-f078-4d5d-b0e1-b878a07ae4f4] err="unable to destroy cgroup paths for cgroup [kubepods pod9c34afa1-f078-4d5d-b0e1-b878a07ae4f4] : Failed to remove paths: map[blkio:/sys/fs/cgroup/blkio/kubepods.slice/kubepods-pod9c34afa1_f078_4d5d_b0e1_b878a07ae4f4.slice cpu:/sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-pod9c34afa1_f078_4d5d_b0e1_b878a07ae4f4.slice cpuacct:/sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-pod9c34afa1_f078_4d5d_b0e1_b878a07ae4f4.slice cpuset:/sys/fs/cgroup/cpuset/kubepods.slice/kubepods-pod9c34afa1_f078_4d5d_b0e1_b878a07ae4f4.slice devices:/sys/fs/cgroup/devices/kubepods.slice/kubepods-pod9c34afa1_f078_4d5d_b0e1_b878a07ae4f4.slice freezer:/sys/fs/cgroup/freezer/kubepods.slice/kubepods-pod9c34afa1_f078_4d5d_b0e1_b878a07ae4f4.slice hugetlb:/sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-pod9c34afa1_f078_4d5d_b0e1_b878a07ae4f4.slice memory:/sys/fs/cgroup/memory/kubepods.slice/kubepods-pod9c34afa1_f078_4d5d_b0e1_b878a07ae4f4.slice net_cls:/sys/fs/cgroup/net_cls,net_prio/kubepods.slice/kubepods-pod9c34afa1_f078_4d5d_b0e1_b878a07ae4f4.slice net_prio:/sys/fs/cgroup/net_cls,net_prio/kubepods.slice/kubepods-pod9c34afa1_f078_4d5d_b0e1_b878a07ae4f4.slice perf_event:/sys/fs/cgroup/perf_event/kubepods.slice/kubepods-pod9c34afa1_f078_4d5d_b0e1_b878a07ae4f4.slice pids:/sys/fs/cgroup/pids/kubepods.slice/kubepods-pod9c34afa1_f078_4d5d_b0e1_b878a07ae4f4.slice systemd:/sys/fs/cgroup/systemd/kubepods.slice/kubepods-pod9c34afa1_f078_4d5d_b0e1_b878a07ae4f4.slice]"
May 14 18:43:20.561933 ci-ln-9ciwnj2-002ac-mdjkv-master-1 hyperkube[1589]: E0514 18:43:20.561898    1589 cadvisor_stats_provider.go:147] "Unable to fetch pod log stats" err="open /var/log/pods/openshift-kube-scheduler_installer-6-ci-ln-9ciwnj2-002ac-mdjkv-master-1_9c34afa1-f078-4d5d-b0e1-b878a07ae4f4: no such file or directory" pod="openshift-kube-scheduler/installer-6-ci-ln-9ciwnj2-002ac-mdjkv-master-1"
May 14 18:43:22.150062 ci-ln-9ciwnj2-002ac-mdjkv-master-1 hyperkube[1589]: time="2021-05-14T18:43:22Z" level=warning msg="Failed to remove cgroup (will retry)" error="rmdir /sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-pod9c34afa1_f078_4d5d_b0e1_b878a07ae4f4.slice/crio-f7ecc406158d24cae9120f43c454d313a4abdee345ebb7c3d89630b97d0d13ec.scope: device or resource busy"

I don't think there's anything in CI-search to distinguish this from bug 1959495, but here's how often we have operators stuck on ContainerCreating:

$ w3m -dump -cols 200 'https://search.ci.openshift.org/?maxAge=96h&type=junit&search=kube-apiserver.*InstallerPodContainerWaiting_ContainerCreating' | grep 'failures match' | sort
periodic-ci-openshift-release-master-ci-4.7-upgrade-from-stable-4.6-e2e-ovirt-upgrade (all) - 8 runs, 50% failed, 100% of failures match = 50% impact
periodic-ci-openshift-release-master-ci-4.7-upgrade-from-stable-4.6-e2e-vsphere-upgrade (all) - 4 runs, 25% failed, 100% of failures match = 25% impact
periodic-ci-openshift-release-master-ci-4.9-e2e-gcp (all) - 75 runs, 100% failed, 1% of failures match = 1% impact
periodic-ci-openshift-release-master-nightly-4.8-e2e-aws-canary (all) - 8 runs, 13% failed, 100% of failures match = 13% impact
pull-ci-openshift-cluster-etcd-operator-master-e2e-gcp-disruptive-ovn (all) - 14 runs, 79% failed, 9% of failures match = 7% impact
pull-ci-openshift-cluster-network-operator-master-e2e-ovn-hybrid-step-registry (all) - 35 runs, 51% failed, 6% of failures match = 3% impact
pull-ci-openshift-kubernetes-master-e2e-gcp-upgrade (all) - 40 runs, 65% failed, 4% of failures match = 3% impact
pull-ci-openshift-ovn-kubernetes-master-e2e-metal-ipi-ovn-dualstack (all) - 38 runs, 61% failed, 4% of failures match = 3% impact
pull-ci-openshift-ovn-kubernetes-master-e2e-openstack-ovn (all) - 34 runs, 85% failed, 3% of failures match = 3% impact
rehearse-17445-pull-ci-openshift-installer-master-e2e-metal-ipi-ovn-dualstack-trigger (all) - 5 runs, 80% failed, 25% of failures match = 20% impact

[1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-launch-azure/1393223397898457088
Comment 2 Elana Hashman 2021-06-02 22:35:48 UTC 
Kubelet messages look like 1952224 which 1959495 is also a dupe of. Having stared at this a while, I believe these are the same issue.

*** This bug has been marked as a duplicate of bug 1952224 ***