Issue is with below builds : 4.9.0-0.nightly-ppc64le-2021-08-17-145337 4.9.0-0.nightly-ppc64le-2021-08-19-120135 on bastion : # lscpu Architecture: ppc64le Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 8 Core(s) per socket: 1 Socket(s): 1 NUMA node(s): 1 Model: 2.3 (pvr 004e 0203) Model name: POWER9 (architected), altivec supported Hypervisor vendor: pHyp Virtualization type: para L1d cache: 32K L1i cache: 32K NUMA node0 CPU(s): 0-7 Physical sockets: 2 Physical chips: 1 Physical cores/chip: 10 [core@master-0 ~]$ lscpu Architecture: ppc64le Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 8 Core(s) per socket: 1 Socket(s): 1 NUMA node(s): 1 Model: 2.3 (pvr 004e 0203) Model name: POWER9 (architected), altivec supported Hypervisor vendor: pHyp Virtualization type: para L1d cache: 32K L1i cache: 32K NUMA node0 CPU(s): 0-7 No workload was deployed on the cluster. # oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.9.0-0.nightly-ppc64le-2021-08-17-145337 False False True 4d5h OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.pravin-49proxy.redhat.com/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)... baremetal 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h cloud-controller-manager 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h cloud-credential 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h cluster-autoscaler 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h config-operator 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h console 4.9.0-0.nightly-ppc64le-2021-08-17-145337 False False False 4d6h RouteHealthAvailable: console route is not admitted csi-snapshot-controller 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True True False 3d21h Progressing: Waiting for Deployment to deploy csi-snapshot-controller pods dns 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h etcd 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h image-registry 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d15h ingress 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h insights 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h kube-apiserver 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False True 6d16h NodeControllerDegraded: The master nodes not ready: node "master-2" not ready since 2021-08-23 09:51:21 +0000 UTC because KubeletNotReady (container runtime is down)... kube-controller-manager 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False True 6d16h StaticPodsDegraded: pod/kube-controller-manager-master-0 container "kube-controller-manager" is waiting: CreateContainerError: open /var/run/containers/storage/overlay-layers/.tmp-mountpoints.json632783936: no space left on device... kube-scheduler 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False True 6d16h StaticPodsDegraded: pod/openshift-kube-scheduler-master-0 container "kube-scheduler" is waiting: CreateContainerError: open /var/run/containers/storage/overlay-layers/.tmp-mountpoints.json574670570: no space left on device... kube-storage-version-migrator 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 4d10h machine-api 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h machine-approver 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h machine-config 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 4d5h marketplace 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h monitoring 4.9.0-0.nightly-ppc64le-2021-08-17-145337 False True True 4d22h Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error. network 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True True True 6d16h DaemonSet "openshift-sdn/sdn" rollout is not making progress - last change 2021-08-20T11:46:25Z node-tuning 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h openshift-apiserver 4.9.0-0.nightly-ppc64le-2021-08-17-145337 False False False 18h APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.129.0.84:8443/apis/apps.openshift.io/v1: bad status from https://10.129.0.84:8443/apis/apps.openshift.io/v1: 401... openshift-controller-manager 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h openshift-samples 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h operator-lifecycle-manager 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h operator-lifecycle-manager-catalog 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h operator-lifecycle-manager-packageserver 4.9.0-0.nightly-ppc64le-2021-08-17-145337 False True False 3d21h ClusterServiceVersion openshift-operator-lifecycle-manager/packageserver observed in phase Failed with reason: InstallCheckFailed, message: install timeout service-ca 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h storage 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h # oc get nodes NAME STATUS ROLES AGE VERSION master-0 Ready master 6d17h v1.22.0-rc.0+3dfed96 master-1 Ready master 6d17h v1.22.0-rc.0+3dfed96 master-2 Ready master 6d17h v1.22.0-rc.0+3dfed96 worker-0 Ready worker 6d17h v1.22.0-rc.0+3dfed96 worker-1 Ready worker 6d17h v1.22.0-rc.0+3dfed96 # oc adm top nodes Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io) # oc get pods -A -o wide | grep -v "Running" | grep -v "Completed" NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nfs-provisioner nfs-client-provisioner-84766f5b6d-6pscv 0/1 CreateContainerError 0 (4d19h ago) 6d18h 10.128.2.9 worker-1 <none> <none> openshift-apiserver apiserver-9fc69858-9tkhg 1/2 CreateContainerError 11 (43h ago) 6d17h 10.128.0.47 master-0 <none> <none> openshift-apiserver apiserver-9fc69858-p9x8t 1/2 CreateContainerError 6 (25h ago) 6d18h 10.130.0.32 master-2 <none> <none> openshift-authentication-operator authentication-operator-656f485cc6-8rtk8 0/1 CreateContainerError 6 (3d23h ago) 4d8h 10.130.0.62 master-2 <none> <none> openshift-authentication oauth-openshift-54bcbd5487-nhkcv 0/1 CreateContainerError 1 (44h ago) 4d8h 10.130.0.70 master-2 <none> <none> openshift-cluster-storage-operator csi-snapshot-controller-74cbcb446f-cf572 0/1 CreateContainerError 6 (3d23h ago) 4d8h 10.130.0.60 master-2 <none> <none> openshift-cluster-storage-operator csi-snapshot-controller-74cbcb446f-hz6wr 0/1 CreateContainerError 80 (3d23h ago) 6d18h 10.128.0.4 master-0 <none> <none> openshift-cluster-version cluster-version-operator-cdd5f5c56-t6d2j 0/1 CreateContainerError 7 (43h ago) 6d18h 9.47.87.104 master-0 <none> <none> openshift-config-operator openshift-config-operator-58684b8cc5-7k658 0/1 CreateContainerError 6 (43h ago) 4d8h 10.128.0.69 master-0 <none> <none> openshift-console-operator console-operator-7b4ddbcf8d-2h4vw 0/1 CreateContainerError 53 (4d8h ago) 6d18h 10.130.0.22 master-2 <none> <none> openshift-console console-649f685c96-g7h96 0/1 CreateContainerError 4 (44h ago) 6d18h 10.130.0.24 master-2 <none> <none> openshift-console console-649f685c96-jzgx7 0/1 CreateContainerError 3 (43h ago) 6d18h 10.128.0.39 master-0 <none> <none> openshift-image-registry image-pruner-27156960--1-pwn2z 0/1 ContainerCreating 0 4d10h <none> worker-1 <none> <none> openshift-kube-apiserver-operator kube-apiserver-operator-54cbc7b859-7jg5w 0/1 CreateContainerError 2 (25h ago) 4d8h 10.130.0.68 master-2 <none> <none> openshift-kube-apiserver kube-apiserver-master-0 4/5 CreateContainerError 13 (43h ago) 5d18h 9.47.87.104 master-0 <none> <none> openshift-kube-apiserver kube-apiserver-master-2 4/5 CreateContainerError 6 (4d7h ago) 5d18h 9.47.87.106 master-2 <none> <none> openshift-kube-controller-manager kube-controller-manager-master-0 3/4 CreateContainerError 88 (3d23h ago) 6d18h 9.47.87.104 master-0 <none> <none> openshift-kube-controller-manager kube-controller-manager-master-1 2/4 CreateContainerError 108 (4d ago) 6d18h 9.47.87.110 master-1 <none> <none> openshift-kube-controller-manager kube-controller-manager-master-2 3/4 CreateContainerError 74 (4d7h ago) 6d18h 9.47.87.106 master-2 <none> <none> openshift-kube-scheduler openshift-kube-scheduler-master-0 2/3 CreateContainerError 67 (2d10h ago) 6d18h 9.47.87.104 master-0 <none> <none> openshift-kube-scheduler openshift-kube-scheduler-master-1 2/3 CreateContainerError 59 (4d3h ago) 6d18h 9.47.87.110 master-1 <none> <none> openshift-marketplace certified-operators-k7r4p 0/1 ImageInspectError 0 (5d1h ago) 5d2h 10.128.3.108 worker-1 <none> <none> openshift-marketplace certified-operators-l7gv2 0/1 ImageInspectError 0 5d1h 10.128.3.126 worker-1 <none> <none> openshift-marketplace community-operators-656xf 0/1 ImageInspectError 0 (5d1h ago) 5d1h 10.128.3.118 worker-1 <none> <none> openshift-marketplace community-operators-znvc2 0/1 ImageInspectError 0 5d1h 10.128.3.127 worker-1 <none> <none> openshift-marketplace marketplace-operator-54c6798b65-77vwn 0/1 CreateContainerError 6 (3d23h ago) 4d8h 10.128.0.62 master-0 <none> <none> openshift-marketplace redhat-marketplace-h4rdg 0/1 ContainerCreating 0 5d1h <none> worker-1 <none> <none> openshift-marketplace redhat-marketplace-zk6tn 0/1 ImageInspectError 0 (5d1h ago) 6d14h 10.128.2.87 worker-1 <none> <none> openshift-marketplace redhat-operators-9k5dj 0/1 ImageInspectError 0 (5d1h ago) 5d1h 10.128.3.116 worker-1 <none> <none> openshift-marketplace redhat-operators-9ldp7 0/1 ImageInspectError 0 5d1h 10.128.3.124 worker-1 <none> <none> openshift-monitoring alertmanager-main-1 0/5 ContainerCreating 0 4d7h <none> worker-1 <none> <none> openshift-monitoring grafana-8657cf6659-dx97s 0/2 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-monitoring kube-state-metrics-7b6988cf78-7x8sw 0/3 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-monitoring openshift-state-metrics-5785f48976-nlzv5 0/3 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-monitoring prometheus-adapter-5f6ff5849b-r79hm 0/1 ContainerCreating 0 4d6h <none> worker-0 <none> <none> openshift-monitoring prometheus-adapter-6bcc5ddf97-krzsn 0/1 Pending 0 3d23h <none> <none> <none> <none> openshift-monitoring prometheus-adapter-6bcc5ddf97-qq7df 0/1 ContainerCreating 0 3d23h <none> worker-1 <none> <none> openshift-monitoring prometheus-k8s-0 6/7 CreateContainerError 0 (3d14h ago) 4d7h 10.131.0.249 worker-0 <none> <none> openshift-monitoring prometheus-k8s-1 0/7 Init:0/1 0 4d7h <none> worker-1 <none> <none> openshift-monitoring telemeter-client-7b954d746b-fspb5 0/3 ContainerCreating 0 4d9h <none> worker-1 <none> <none> openshift-network-diagnostics network-check-source-dfbc58d6f-mb2b5 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-oauth-apiserver apiserver-6d5b547944-r2njv 0/1 CreateContainerError 3 (45h ago) 4d8h 10.129.0.85 master-1 <none> <none> openshift-oauth-apiserver apiserver-6d5b547944-zqnh6 0/1 CreateContainerError 7 (4d7h ago) 6d18h 10.130.0.9 master-2 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156585--1-7gnj8 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156600--1-bfwnf 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156615--1-hvhzf 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156630--1-bm8dh 0/1 ContainerCreating 0 4d16h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156645--1-nqcsj 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156660--1-gckl4 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156675--1-d2hvq 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156690--1-n6pbn 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156705--1-flx77 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156720--1-ml56f 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156735--1-6brkc 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156750--1-9q4k2 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156765--1-kwch8 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156780--1-8vkj5 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156795--1-qg8fw 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156810--1-hz9mm 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156825--1-w2wvk 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156840--1-mv2fd 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156855--1-fpgrn 0/1 ContainerCreating 0 4d12h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156870--1-6c7ck 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156885--1-z58sv 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156900--1-q885s 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156915--1-78mhb 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156930--1-898zx 0/1 ContainerCreating 0 4d11h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156945--1-5grj9 0/1 ContainerCreating 0 4d11h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156960--1-q77nv 0/1 ContainerCreating 0 4d10h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156975--1-tqr9j 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27156990--1-87mdj 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157005--1-rcr7m 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157020--1-x6jdv 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157035--1-5mhjc 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157050--1-8nbjz 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157065--1-xfvms 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157080--1-g5f9q 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157095--1-tcch5 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157110--1-jh8x5 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157125--1-lfcl4 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157140--1-pfb9r 0/1 ContainerCreating 0 4d7h <none> worker-1 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157185--1-c99ph 0/1 ContainerCreating 0 4d7h <none> worker-0 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157200--1-vc9kr 0/1 ContainerCreating 0 4d6h <none> worker-0 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157215--1-7qxjj 0/1 ContainerCreating 0 4d6h <none> worker-0 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157230--1-bmr9s 0/1 ContainerCreating 0 4d6h <none> worker-0 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157245--1-8sjsd 0/1 ContainerCreating 0 4d6h <none> worker-0 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157260--1-vrchv 0/1 ContainerCreating 0 4d5h <none> worker-0 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157275--1-tdq9c 0/1 ContainerCreating 0 4d5h <none> worker-0 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157290--1-v782d 0/1 ContainerCreating 0 4d5h <none> worker-0 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157305--1-5d5zb 0/1 ContainerCreating 0 4d5h <none> worker-0 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157320--1-jwh24 0/1 ContainerCreating 0 4d4h <none> worker-0 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157335--1-gltgf 0/1 ContainerCreating 0 4d4h <none> worker-0 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157350--1-jbmhc 0/1 ContainerCreating 0 4d4h <none> worker-0 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157365--1-jbmn8 0/1 ContainerCreating 0 4d4h <none> worker-0 <none> <none> openshift-operator-lifecycle-manager collect-profiles-27157665--1-4spth 0/1 ContainerCreating 0 3d23h <none> worker-0 <none> <none> openshift-operator-lifecycle-manager olm-operator-75dcc9cb6-8nt5n 0/1 CreateContainerError 4 (43h ago) 4d8h 10.128.0.65 master-0 <none> <none> openshift-operator-lifecycle-manager package-server-manager-7b7646568c-gkx7k 0/1 CreateContainerError 6 (4d7h ago) 4d8h 10.130.0.55 master-2 <none> <none> openshift-operator-lifecycle-manager packageserver-76694fb586-9pph6 0/1 CreateContainerError 7 (43h ago) 4d8h 10.128.0.68 master-0 <none> <none>
[root@master-0 ~]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 7.9G 0 7.9G 0% /dev tmpfs 8.0G 256K 8.0G 1% /dev/shm tmpfs 8.0G 7.9G 151M 99% /run tmpfs 8.0G 0 8.0G 0% /sys/fs/cgroup /dev/sda4 120G 17G 104G 14% /sysroot tmpfs 8.0G 64K 8.0G 1% /tmp /dev/sdb3 364M 233M 109M 69% /boot overlay 8.0G 7.9G 151M 99% /etc/NetworkManager/systemConnectionsMerged tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/bdfb85ba-24e7-43ce-b9c3-5c60c3c9669a/volumes/kubernetes.io~projected/kube-api-access-f6vv7 tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/11ccb899-73cb-489c-b8f3-aa33a2625c8d/volumes/kubernetes.io~projected/kube-api-access-lzq5s tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/d146ea13-e8ce-49bc-85ad-57a5ed0e652d/volumes/kubernetes.io~projected/kube-api-access-xgvcn shm 64M 0 64M 0% /run/containers/storage/overlay-containers/d377af6e04a58c9a80c8c0717342a303c08670c5e0cc0f41328ad6b2ddd298ad/userdata/shm shm 64M 0 64M 0% /run/containers/storage/overlay-containers/363a5a988d9fc0c1529bc651dc81b34ed0c388a936a92039467280db1d1be88f/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/44485d567266773853d769b527885816b8b3a1a977286f4c2856256859e26fa0/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/4ebed2b3fdc214373aaedbfde7aadf3e78e115090fbc5233dc4de5a9bccd514b/merged tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/159aa25b-c750-4b2b-ad2e-10693ca53127/volumes/kubernetes.io~projected/kube-api-access-2fb9x tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/7bfa396e-3ea9-4c20-8b43-a3bb51f8f641/volumes/kubernetes.io~projected/kube-api-access-25d8k shm 64M 0 64M 0% /run/containers/storage/overlay-containers/955e2b9b0ce36cd4738e347131beb0bf7be9423947e5712e3c402885be53d909/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/b766b0cedf2fef1f27d71b92a3adf0e986a94d10051c7b82fe49322430afc195/merged shm 64M 0 64M 0% /run/containers/storage/overlay-containers/5d5ba79f040f6afc2ba582db6dbdfbcc7edd2992fd4227932c056073d1daa890/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/4c76b73d982a7f0521ab0c01817e163332646d11b3d095abc5109d15a9925a4a/merged tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/71b947dd-2538-4a51-85e1-574d95737d86/volumes/kubernetes.io~projected/kube-api-access-fnxrk overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/23ffe731d98acf25c11542241bbd32a3414007a1720eebd5c459bdf02a6eceda/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/bffdefd657991da2d0e371c503b15b8108dc4678a0d43bbd59456432950fb370/merged tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/9eaacf53-65e7-40e7-b2dd-a63a02628189/volumes/kubernetes.io~projected/kube-api-access-xvqzg tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/c8ce35cb-071d-492c-8d17-99a4dae7fe31/volumes/kubernetes.io~secret/sdn-metrics-certs tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/c8ce35cb-071d-492c-8d17-99a4dae7fe31/volumes/kubernetes.io~projected/kube-api-access-6zrff shm 64M 0 64M 0% /run/containers/storage/overlay-containers/3461bcfae511eb2b540880ec881646a1a06cf54f68861646e5b24374244674d8/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/a4261b0978f774be727ce36dd694c17e4cca581ac83d853673fd9a969ddcbc50/merged shm 64M 0 64M 0% /run/containers/storage/overlay-containers/246eb46b07a6bf7a79e39f4d0627c1c716357079a1a3e4c061f64470b17b10b3/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/d74089f8d7ed55080b577a5db1f43acc63cb812ad4824ede346beb3292475e98/merged tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/35585c2b-1aaa-47c4-b1ce-c646acedeacb/volumes/kubernetes.io~projected/kube-api-access-4qqr8 overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/1455f1c77d1567a5575d00afb8b87c2296afcb5c4ad56be0dc8e58bfbf82d05c/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/0fe411f8ffda89a751bfd9f67ac3889c190d5fa2438a8b99a6c60ea33246ba66/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/8437c2dba678b402c382d7df5b81d182b1a3878b883d266f937dd22917c43ed9/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/7b12e1c6436da8d8763227fe649c585b1912b89002b762a50770463ecdf12342/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/28754d2d25cd9e871393f628afaba58936902e0cade9feb0a319ae46d747523d/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/14c1aa138f1015a7e13c54ccd007eba1cc6f3c17e6d77906e864f2cbf7adfe4d/merged shm 64M 0 64M 0% /run/containers/storage/overlay-containers/11ad5783105b25cc88b4bb35a9f70562f584503e5397dc44ad198d871d95882a/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/577520db1ce12b1d4124d3c46cb6621aecd772ef9143a1ed69da059b2538e69d/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/a702f92f7c42a1596c20808bb8d5562386c938cd166e2126fc9824bbeb626115/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/8b06a88b188b2917a7906a6161c289e200fc50c6b591033cf618e4b202a87575/merged tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/52460172-1449-44b1-939d-66c45d794d29/volumes/kubernetes.io~projected/kube-api-access-kxccz tmpfs 15G 64K 15G 1% /var/lib/kubelet/pods/1462887a-d5e4-4351-a25f-819261fdc0fa/volumes/kubernetes.io~secret/cookie-secret tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/1462887a-d5e4-4351-a25f-819261fdc0fa/volumes/kubernetes.io~projected/kube-api-access-qncz9 tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/597b3475-78e2-45eb-9ff5-58f1d636cdf2/volumes/kubernetes.io~projected/kube-api-access-m2vbm tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/264027c1-88e3-45f4-ab59-00dcf1ea23a9/volumes/kubernetes.io~projected/kube-api-access-s4stb shm 64M 0 64M 0% /run/containers/storage/overlay-containers/87b10f0163ad881a169de41bb2368618cb065aa973fb0017e3e64df7babaf07b/userdata/shm tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/cf31af7e-20e7-4bfc-b72f-03490778d033/volumes/kubernetes.io~projected/kube-api-access-9r7zp overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/7c9a09928efd9fd182251d6d99cba1857bbac10a6cc7a06eba372714560b7318/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/422074682e6ce253cc460e576e20d9eda6f8e8993a091411b6ef9fc1e4d41519/merged tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/8b7a583a-1708-47b2-9b0b-a6f6e8654f3c/volumes/kubernetes.io~projected/kube-api-access-9749n shm 64M 0 64M 0% /run/containers/storage/overlay-containers/4ee7a4973c861b3818d29968d7458ec5a3441116ee1e773bef60cccaa8299ae0/userdata/shm tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/b7b09c30-6c3a-49fd-bd88-32da6920906a/volumes/kubernetes.io~projected/kube-api-access-jps9k overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/8a0a9788f590d99e608bed0765093d96a03d892cfac31a6ee31b786b8b4b072b/merged tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/b7b09c30-6c3a-49fd-bd88-32da6920906a/volumes/kubernetes.io~secret/etcd-client shm 64M 0 64M 0% /run/containers/storage/overlay-containers/be6d99eebca40c444781a1193c4573457780d5daa71af6cebcbe032844f285c1/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/bb32a5a55d7c1dfc870a22b8bd727d6c255e183c505913911d87b258e29ae9db/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/1e2c6944613074b1c0337577e368b94fd23f9c745c69aac68fae550b81e7beec/merged tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/597b3475-78e2-45eb-9ff5-58f1d636cdf2/volumes/kubernetes.io~secret/prometheus-operator-tls shm 64M 0 64M 0% /run/containers/storage/overlay-containers/2d176679f11fe94476ae277fa98bad58f8bfe88385c8eb944b7e12c4172d8140/userdata/shm tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/cf31af7e-20e7-4bfc-b72f-03490778d033/volumes/kubernetes.io~secret/certs tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/52460172-1449-44b1-939d-66c45d794d29/volumes/kubernetes.io~secret/webhook-certs overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/4364916568d74d487f7e3db22de0060fe1dcbee8103c6a5fd121772b286cac7a/merged shm 64M 0 64M 0% /run/containers/storage/overlay-containers/b718177cdbf4f36b41309f4665f05b68e0b7488674b15627bfcb5420ba1ac209/userdata/shm shm 64M 0 64M 0% /run/containers/storage/overlay-containers/e6d0c8d7bad2960d6a75ade4cf388fd9a9d87e49f7d9f9a97b20647728021bc3/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/9ed437034cf01ed2e7041420da5542172090f85e01473559ac6873a5d3495aa2/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/bb2e8160fe78fbceb70842fb57449d684dba2dbce7928b4b5b64b76642b42af5/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/2191129c6eb07104d7a7f269379aa79d8d31195b6f8c56b3d6af78751a1d1d4b/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/5b087f59a8f978138c6b2001d4bf7ed22eb89458d1e589efd3faebd9a054384c/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/8435500c0409cf38c65cb328faf91824dfcfe156763b995c22536bb0a9e72b5a/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/0c8199464e8f972622e18882ffba70384e021a4326588ecf447c1346cf803403/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/c45e91019b8c7fb327459b225c9fa547854673eacd196faa1760434303b15d67/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/604a99e0ee98836d187486cefc176f6bf380d480bbcb2ac2875d849ed16f1bbe/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/b8c71b49999a9a5aa40e88d73d2d39cf2900b0f59e0828d1103b92767251d14c/merged tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/1462887a-d5e4-4351-a25f-819261fdc0fa/volumes/kubernetes.io~secret/proxy-tls shm 64M 0 64M 0% /run/containers/storage/overlay-containers/71741a1d015090c170e594efc313d3d56e015e4453a619391e0422d4ba391989/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/6d82dfecadb4e2b251d17a48ba4ac8c2228bbe268b5d2e7792c3ab9eb9ea8645/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/4bd51cd1bf5f613031766f6292c0dde5832f647de3ded4a5c5a5987baeddad3f/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/debe01bb2b1d22fe17e05173a224771da58b14dd045e69e11bf16c38d839305f/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/f68f096d0f26ca96a06837a40b9c007da32834c2aa9992a5a323ed768b041af6/merged tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/e6ca5d21-13fe-40f3-ae7d-2e39f55d39d8/volumes/kubernetes.io~projected/kube-api-access-r9ckd tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/e6ca5d21-13fe-40f3-ae7d-2e39f55d39d8/volumes/kubernetes.io~secret/node-exporter-tls tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/d146ea13-e8ce-49bc-85ad-57a5ed0e652d/volumes/kubernetes.io~secret/serving-cert shm 64M 0 64M 0% /run/containers/storage/overlay-containers/2c88568acd46744465e17ea914036b9f4e6eebb304f6113c6bfdc8dd47ea2ef1/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/9b51e4f3823fb2d1089bbde721293076babd7f75bd28ee16f4ae7b801278acd1/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/dc31f2fe7b1f0ca070d7af0a91e78534f0cced83d5a7527235bc979a2af8af80/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/deef2458a3f83a4e04cb7eb466300baffce5c146301ac02e1c9b4b38cd5c61bc/merged shm 64M 0 64M 0% /run/containers/storage/overlay-containers/d3bb24a744d9a6a210c778990b7943a836571fe1a3300a9a70ff358a67d5c151/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/68c4044d8640bddc50ad549846328068cfd8a7411d0107cd70b1ed29255cb053/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/eeac4539b76bf78ad5292d365a97d1fedf8b1975c5f914bbaffa688673a016cd/merged tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/5144c896-68ec-49ef-970b-0dfe93b07230/volumes/kubernetes.io~projected/kube-api-access-w8qkj overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/2eb2e3f3c6d16c01e2e14f50fc583cd85e5959ce168af4c964608dfb7ee1a7d2/merged tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/71b947dd-2538-4a51-85e1-574d95737d86/volumes/kubernetes.io~secret/metrics-certs shm 64M 0 64M 0% /run/containers/storage/overlay-containers/16ac7eff58385fa96458f451a7ad3c78554348e61eb1f77bd53004303a523073/userdata/shm tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/035193f0-e425-43c5-8731-0a0dc3198ff2/volumes/kubernetes.io~projected/kube-api-access-vhj8q tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/712354da-4ba0-41bf-9204-1d73fb56b0c4/volumes/kubernetes.io~projected/kube-api-access-sx9sd tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/035193f0-e425-43c5-8731-0a0dc3198ff2/volumes/kubernetes.io~secret/metrics-tls tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/9a904603-969a-453a-b253-0260499e548c/volumes/kubernetes.io~projected/kube-api-access-vkzrl overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/f6094266644a3c56fcc03bac5084971a348bd8d9be295091c4994eb65219ccd5/merged shm 64M 0 64M 0% /run/containers/storage/overlay-containers/1fdee263aa710d35172ba292d7469b043a2b4aa7ee5b472346c368a09b25539e/userdata/shm shm 64M 0 64M 0% /run/containers/storage/overlay-containers/0d4ea08e8d3ba8514d012cab77a5a8c5456ff80b08ab22cc0bba0fa53ed82db4/userdata/shm shm 64M 0 64M 0% /run/containers/storage/overlay-containers/fb1ae63b3c9bb9d53658cf345cabab0a7aa648d7d22d20b4cc2b7707b15f5be5/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/ba23bb184d211c6fa1c19d6b4cd7e02181856fd8484a0ce2bfa1ed21716e6434/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/d92df05677a5c021db8b78e379a5f4643668d0d29dd0e574829f67521ce8e686/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/3ddc54f4966c180ef02eb3d24bbf4b738bb6686f7bd2e78720ff04fc5dcbdccc/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/6615919ae31a4694aacd70e4e3764b0bcf20f3eb4f5d93947d58dc2596acdb1b/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/2a34aeefd85fd4bedab705b93b2cfec630afb8b695bb2ba8c046af23ff530326/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/ee195cb8d6d4ce5a6bbaed93dc0f6c716f2f3742b3766eee9456187734c23c38/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/54d1307916a3ee47428261637f4b3e019f401a3e25b7dbe996b9266037ea1974/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/643e54d48ee05b1a990346b604d02c98157607727082f4049e43bc32d39f1258/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/af8e8df2f230ff92e06ac5bf53f0fd4fcf4edc079e729c0966aa203e1cf598f6/merged tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/648bf02a-db73-402e-a0ee-75a96c9320c0/volumes/kubernetes.io~secret/certs tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/648bf02a-db73-402e-a0ee-75a96c9320c0/volumes/kubernetes.io~secret/node-bootstrap-token tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/648bf02a-db73-402e-a0ee-75a96c9320c0/volumes/kubernetes.io~projected/kube-api-access-qqvbj shm 64M 0 64M 0% /run/containers/storage/overlay-containers/89d79d70c51d54bbecb286738e307367ed4108ebac79dad3050b0bdd7699243b/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/9d2caacd02661d4f1f9523c986079fd0810f519c61af19a91397beef4af198d7/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/f2fbb5210c3ac090d5837ade1b6a38340041f72c01927f916f71e9fc9577d61a/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/512a966052bd0bb06aeda04e6f3d99d6866af9347b0e93b0048c41ecc194a557/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/e8fb605cb6e86d6bd0430fd86746d942e8460fbf474db898feffa480c7b3a9c3/merged tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/3cf6965e-984f-4bd3-8339-bf04dee44825/volumes/kubernetes.io~secret/etcd-client tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/3cf6965e-984f-4bd3-8339-bf04dee44825/volumes/kubernetes.io~secret/serving-cert tmpfs 15G 0 15G 0% /var/lib/kubelet/pods/3cf6965e-984f-4bd3-8339-bf04dee44825/volumes/kubernetes.io~secret/encryption-config tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/3cf6965e-984f-4bd3-8339-bf04dee44825/volumes/kubernetes.io~projected/kube-api-access-mbvfj shm 64M 0 64M 0% /run/containers/storage/overlay-containers/9c5f3db4be82151e89af5fbb938343596d27f48d2f0f5e4f68b5ee8f70525226/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/2bf1b8dde0c80013bbb25899444c01ea9c42440ebacc0bce98195b573d40f466/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/13553a1709275678b442cc97c8ffcc04c5fedd6c4adf91f25e6fdc23c8467944/merged tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/7a55cabd-0e46-42af-91c6-5c3ba72fccef/volumes/kubernetes.io~projected/kube-api-access-wpv6c shm 64M 0 64M 0% /run/containers/storage/overlay-containers/735b02248120c57cb9972289b180a015160e5ef5e2ecde7efd2879618a436584/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/5347813ed4fd974f910bcee624e98f9da4723fa53de96874c52d63583029163b/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/8fd98f96bab35a0ac36674f17d5d6281a1d0226753fb2e0264df78924d7cdfba/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/354634dfcbdd9babcd9b74106aa21c4118a9fe7586e7829d04e931e5ae5a0c2a/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/079eb8700cb53d61eaaca0d0360a622135717815c8927ed895c68d85e51c73ef/merged shm 64M 0 64M 0% /run/containers/storage/overlay-containers/a42d82272bdbcad9fd0c9effe146e124a5af6799f7024b3f6d85f77ddfe6acbf/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/aed5e2f54a47a1d68241be2e80c1b0d88846c331adc0e34dc9277e25817bd347/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/5c8c934f3618638a8e8dcd5c1c7ff4b1727a4624beff05bc70f2f3a5db92423c/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/4c9b916a67c848721a460f196332ab149d7faab63628525c132a1b1b1beb5224/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/35f1a33abf33efee106f25d7e4800bde8bc5d2702e7dbdfa77c9a4f7f817a910/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/11654a848f69a6bd4510857c3580eb2c1909a784307631b76cf8861691b474e0/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/8fca976b76b7af09cb9f838f29cde5dc0da6923e289ce788064e8dd83d5f1876/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/9e9afaa5b6da0dc036b6795c82626b0842f4ecd36096d03b6620142eabff5567/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/fe3e551263b3eef3a4e09f1e9ed5cdb3d668af1bd35e2a442af40b4e29ed9221/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/07cf66c2a60efa39286569421c95a38841e68b3c378d15ccbf362bdc5b85b05c/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/a4b209a6e3040a8abc5417e41c0276a288a1559fc714e04688bd238a5a4832c9/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/be0df92b202a7efef91e2116b990d6e263534e9f030718291e0eaeee0d6a56b6/merged shm 64M 0 64M 0% /run/containers/storage/overlay-containers/c580c26244ef801b257ffcc6f544ea58d4669c78512192290a47e8cc96834815/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/b169bed621bd457baaeb4c0cfcd27a6cf09c6871996218c64d8d3ba77c2f9473/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/0cc91e6fcadd1f4859b3c1ae02d59257b0230507bb4ba392b28ae0cfb281757a/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/071aa340cdda57c663cbd10bce3a977894694865fc44d28a41db060502837766/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/7aae6e1a76d7d2a548410c262776e9ceb4995172d27bce5021cec6dbaa73763d/merged tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/9a75c7bb-4c75-455d-b005-b25c05ce323d/volumes/kubernetes.io~secret/console-serving-cert tmpfs 15G 64K 15G 1% /var/lib/kubelet/pods/9a75c7bb-4c75-455d-b005-b25c05ce323d/volumes/kubernetes.io~secret/console-oauth-config tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/9a75c7bb-4c75-455d-b005-b25c05ce323d/volumes/kubernetes.io~projected/kube-api-access-ld8d5 shm 64M 0 64M 0% /run/containers/storage/overlay-containers/759b0e66ffed1e82dcd685463b2f8374a7bd79c9e0a79ed562283db59a70fec3/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/73c1174d058df5dd681a08a4f654ac9e005f3d5692f80a2a53b7c56c9b674233/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/78c1177c6f774abab59268776d52b266986eacba73ef96724ce2be5c515f5395/merged shm 64M 0 64M 0% /run/containers/storage/overlay-containers/093958aee6a27b8a55e84a80c4505dc70a7b27a14206f323247a9d0ffa81de57/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/5b6b93aa01981f1baca5febbeafbbb439db2f5ef8990e8c007adfbc028a12590/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/5f70df63e2f732988719a6d014f94b3a328fba37d4a34125cf8af29ff7540a9b/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/340b65a455abca20ebad9d2d00acbfb9a3369bf5dd11582eda33b9acf79b676f/merged tmpfs 1.0G 256K 1.0G 1% /var/lib/kubelet/pods/0acb39bc-71b2-4118-a2b3-0ef163f62651/volumes/kubernetes.io~projected/kube-api-access-jqmq7 shm 64M 0 64M 0% /run/containers/storage/overlay-containers/5b44493129ccc577e465d5f3315dbcb549eb285f5b38c6ba00122896d899a1e4/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/b6b25d8f4d3420b385d3b1fd78a9ff64d5d8b1dfc961f4975eabc57a5ebdf7ae/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/ebaf9e0e02f6bbb3b7f9da9cfa2e081d23ae319bc9c448ac80bb18579959f489/merged tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/74c36eb3-333a-4c8f-af2b-d0a7da1fe0d0/volumes/kubernetes.io~secret/serving-cert tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/74c36eb3-333a-4c8f-af2b-d0a7da1fe0d0/volumes/kubernetes.io~secret/etcd-client tmpfs 15G 0 15G 0% /var/lib/kubelet/pods/74c36eb3-333a-4c8f-af2b-d0a7da1fe0d0/volumes/kubernetes.io~secret/encryption-config tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/74c36eb3-333a-4c8f-af2b-d0a7da1fe0d0/volumes/kubernetes.io~projected/kube-api-access-vsn8m shm 64M 0 64M 0% /run/containers/storage/overlay-containers/5ade87ba95ebde992ac9d67835a7fe5b2b688eb04868cbdf9953d20eb90c2eff/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/b13ab0222ce7323c90dfe4188cd3bdd6749e4449b376771a9f42abe2a0ed2eb8/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/e82fd21305d1ded85aa9c378b0cc69cf72b64583d123d29f03ecac09bc643d2b/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/cf018f9c2f8ec06f3ed6754bcf53dd37930545c616e8e47e7ba930c7cbd14faf/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/b34fd49efabb09b0a1d0887331a31185be4eefdfe56d6ba426d8b9de1fdb6a25/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/71d4a4536df8e88773e46bfc88d8a46c68d564578a8b06cdc4c090903c2902a9/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/4539cb135b738c105561a119031050849090106053f060f9dfebeade2e854092/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/48fc0b5efc0b969d8c6e73fa2b74735a4be116281a454151102dcc6dc98b68f3/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/d3f1162d7db36679117b76f1b1edebbb08eb69396371885f888f361da7dbe977/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/b5005404482c244a78c74dd2e4d44f7f62ae76e9e1d8f72bb07f99815336d059/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/53bb3ffeacffca6c379ed657ae9180c49b2bc2c6a3a57476e8aec706a83d88dd/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/098b8ace46a3364373ee9dbe98697b493e0fc764bbf07d174a0466a238e36fa1/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/b88172b39e73b66b58317026795b65b3e2012f156ac8c6da69f552304efdf951/merged shm 64M 0 64M 0% /run/containers/storage/overlay-containers/0ab1a7f1ac4b092627fa9c9cba2a2f3752aadb3d7773d732fea03e6bd30f839a/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/d76f0ab85843a58d275325e879c062a3cb512db5271ca1262db8f2b9e00b0a95/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/31259c200825f50dcb567b849f49b69423824bc67fe09bfbb5844d1e94b96ca9/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/5ac5a808915a4694e4bbb1c473240fb2372807d40d921eeb1834ad10d9cf4151/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/f9abfe08e56b962467e6258d17e178a6447f7cec169f9375f4271aecd6e981a2/merged tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/b5f720dc-b2ca-49b0-8e06-1431749697f1/volumes/kubernetes.io~secret/serving-cert tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/b5f720dc-b2ca-49b0-8e06-1431749697f1/volumes/kubernetes.io~projected/kube-api-access-kkffr shm 64M 0 64M 0% /run/containers/storage/overlay-containers/0caf934a1460b937014822e12280462212c969e08ced093abed20a19329355ec/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/b699729d831f9ee9434e348c4687779212f9c257c31aa9824b7842a066121626/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/927ac8f2ad1a08775b8ce7ceafcc84d6a3451d43720828340a70e80e74032ed0/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/3f6c16cdb87754d38098dce1a079df83d613db3fc5fb8e77cc7e2e80e77c98ae/merged tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/d1011eaf-7c5e-44e1-8d6e-f7f6aca778b1/volumes/kubernetes.io~secret/serving-cert tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/c8d1e89c-700e-4cd8-bf96-ef453367fe10/volumes/kubernetes.io~secret/profile-collector-cert tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/c8d1e89c-700e-4cd8-bf96-ef453367fe10/volumes/kubernetes.io~secret/srv-cert tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/d1011eaf-7c5e-44e1-8d6e-f7f6aca778b1/volumes/kubernetes.io~projected/kube-api-access-v52w7 tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/6c4bf2b8-581a-4ddd-80f7-79f176570178/volumes/kubernetes.io~secret/webhook-cert tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/6c4bf2b8-581a-4ddd-80f7-79f176570178/volumes/kubernetes.io~secret/apiservice-cert tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/c8d1e89c-700e-4cd8-bf96-ef453367fe10/volumes/kubernetes.io~projected/kube-api-access-4zfg7 tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/6c4bf2b8-581a-4ddd-80f7-79f176570178/volumes/kubernetes.io~projected/kube-api-access-556zl tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/b2b01aaf-ff5e-483d-b75b-260d87a69f42/volumes/kubernetes.io~secret/webhook-cert tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/b2b01aaf-ff5e-483d-b75b-260d87a69f42/volumes/kubernetes.io~secret/apiservice-cert tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/c332f145-cd5b-47a1-bb94-d2b729a59607/volumes/kubernetes.io~secret/signing-key shm 64M 0 64M 0% /run/containers/storage/overlay-containers/c22a60ed3ff34187bde8a71a85ed2f08c76c169f4a53c06566cc801df4dfd9ee/userdata/shm tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/b2b01aaf-ff5e-483d-b75b-260d87a69f42/volumes/kubernetes.io~projected/kube-api-access-78f5d shm 64M 0 64M 0% /run/containers/storage/overlay-containers/47682277b69ee70fc86f3ddda796bb60ce2f3ca9b51eaa133352da9e2281bcd5/userdata/shm shm 64M 0 64M 0% /run/containers/storage/overlay-containers/787daa1f45b018c9347574bac3e88fbcf63bb121f23d20a469f138693ec387d6/userdata/shm shm 64M 0 64M 0% /run/containers/storage/overlay-containers/bfa692aef2c6c01738b8dd3f7e1194aa8a0156b756e2054568893edabb103d1b/userdata/shm tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/c332f145-cd5b-47a1-bb94-d2b729a59607/volumes/kubernetes.io~projected/kube-api-access-76zgn tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/8bd7e035-90db-4f6d-8fd2-61227e103766/volumes/kubernetes.io~secret/marketplace-operator-metrics shm 64M 0 64M 0% /run/containers/storage/overlay-containers/b8708b00fcb37b08e8b0000abb8f3850139386330a786db8037ba27092944d14/userdata/shm tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/8bd7e035-90db-4f6d-8fd2-61227e103766/volumes/kubernetes.io~projected/kube-api-access-shlff tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/6167bb9a-2410-484f-849b-c5f66990140f/volumes/kubernetes.io~secret/serving-cert tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/ec6fe2e5-0c9d-4c68-9246-164ef7f0895c/volumes/kubernetes.io~secret/serving-cert tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/6167bb9a-2410-484f-849b-c5f66990140f/volumes/kubernetes.io~projected/kube-api-access-2zj94 tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/ec6fe2e5-0c9d-4c68-9246-164ef7f0895c/volumes/kubernetes.io~projected/kube-api-access-fqll8 shm 64M 0 64M 0% /run/containers/storage/overlay-containers/f3d7111e93259d8ca4038ddd58397b7b01493d8710a732193bd47708cef7434d/userdata/shm tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/b91d4004-37d3-471a-aae0-78d7cb3e72af/volumes/kubernetes.io~secret/serving-cert shm 64M 0 64M 0% /run/containers/storage/overlay-containers/f809f145032c755036206515e0a79509a9b3cf3bcea9e8a4e32181738394e9c0/userdata/shm tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/b91d4004-37d3-471a-aae0-78d7cb3e72af/volumes/kubernetes.io~projected/kube-api-access-bqx7l shm 64M 0 64M 0% /run/containers/storage/overlay-containers/9d00a4cf82e07c87a87ffc750712b02af62d2e2dcd249ff5e0032c56fb5c92d9/userdata/shm shm 64M 0 64M 0% /run/containers/storage/overlay-containers/62b9a05195d549b414f7f4a03e7682535b6e1b7de76fa2b2e030f03bef752b0d/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/52506c8c3ff7566886a3c35eddabe227cb1a69469aac8620b95c9ea0cc950293/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/2d6acaf2e69f67e58c17edce54cf4223457b3a45bf192e2a0c91dc2fd1cce72c/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/520ed3bca7bf404aa64bd1523c805104068dee7ab92a9c4e8715e482d6080cb3/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/830c3b9fc186ebd3c8a702e35b7945f5c609de701aa6711bb254ef0d596b4d28/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/3692cad9237f6b0b6fa06aac3aa260ec66e3ab21834a208e152d83ae9e8fc0e7/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/a124356d95e87080a26faac125959e371f97bde3a3cded1666c708ef6298142c/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/e0957102e55fd77cc34c93bc4bdf125551fff0eac60feb3a858b535e45cd2332/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/9ec791757c9323f09bd3d696b19cd1abd90eaeb74ced2fde3cae9e9a78ea1903/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/aa99aac8946b20cec123c5208607446f3f90d04cb1f53f3d7ef1dd887ec42017/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/2c6eff585e5883d42e1d439e3ee7208e8cec80678cc9bf0e38e632b271b267db/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/a9c4466aee2b3eb6907502bbbcc3a06411864b8b103bcc0f6c9e05539a13ad10/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/f56e57d4e17b253d83a026595d2b8c6095165eab117f834269a94dcc47638cb8/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/993b2e1d1294bb735ab2deb5aca534c35a60a19188e09f8319448b8e7ee12589/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/bb57e0fcfc9b7665a267c02ec45ea3fa2a6b16c8faa7729a0ee96e1a2994669e/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/8e6515b67a55944323dfa7907cbc7550d2fb50a5bcd6f3c034f7ce80d3de79f0/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/fbb08bc213a2dc72b79480e9a7a1745fe5f1efa04639c8c611b69dc447abc29e/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/f7ca051fab2b840043f08cb0c7e05f469e6919c16d227e744e08f0a4f0e91098/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/e424b81a00ff7b12b71be4c562dac0654113c745379bd2dbe80e0f012af3ecd2/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/79da8b4b992376d0dd849b44ffb5cb1b3e776f9d7b977f3a04c4b5dd76268639/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/8aabf43ec00ec0fe5fb120118025f2032f15831ff1714c01cf4201cd5d1a4a54/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/95ea3c8e78eadd700c696671d9af8d4cec51775edd0bcddea6f33d1922828b5c/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/2b21c6bee3db63d3cf16b3b46df1d905f4f1de190e464bac42e460f699c190c1/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/eb881a78b814e685e522d9ebae8c361592afd7295f46b63683229ef8046abba7/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/5a06cc237eff175fb35c865a712ad37703a45efa06bc5dbf2864f65fe9ec8e89/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/f1bf513a30b72618f8601345e860065738b47fca5ddb437451fa162ff3910795/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/ea3e35b476f5c0089e101349709e2407372890e17eb2df9cda44f9a2e3fb0d12/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/d3e0b171c50cc3b36188ec1f2254b685273863a266a9896bdd6737b4aeb95deb/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/2067cdfdf2418875272767d34a2a89774da507e368de5ddf1d12680c15c75d61/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/34133c03e66c406f77e81de06c1bef4cb8bb233a3b848ef9d18aa6513c1f1ba6/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/e32b8e71dbd9979aca5e986910621ce7e0f7b80c0a145c0c692598f4f2e8c067/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/504caa0b1748deea5a9173b1942f896fa53cfd4230616aa589651daecaeaed9b/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/c5c76564e394347a617ac900ed6b79de35d0b4e5f0f338a94036ac6ef268a2d4/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/cc3540cdef68e4c27495fab763d62fc0880a976486ad8f586eb4d51fb9a198cf/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/6006da855bf8ed01698c949d9e27a943cb0843e1a076dc3ae300ff305c506dda/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/a5ea18ecb340a8bf875f4b02e78a05b094e078dde063c98bb4bf072b9ea8e4c8/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/aebfd1a7960854fd0337e7c1cd2eaa5b42051d4756257e7c91826e8b6d2010c6/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/eb9b0f76a9452a42a36ab9abc7681005fc46ee871ea14e6aefa9c8dfd7dd6878/merged tmpfs 15G 128K 15G 1% /var/lib/kubelet/pods/0f48beb7-e568-40fb-804e-f63444ccefe2/volumes/kubernetes.io~secret/v4-0-config-system-serving-cert tmpfs 15G 0 15G 0% /var/lib/kubelet/pods/0f48beb7-e568-40fb-804e-f63444ccefe2/volumes/kubernetes.io~secret/v4-0-config-user-template-error tmpfs 15G 64K 15G 1% /var/lib/kubelet/pods/0f48beb7-e568-40fb-804e-f63444ccefe2/volumes/kubernetes.io~secret/v4-0-config-system-router-certs tmpfs 15G 64K 15G 1% /var/lib/kubelet/pods/0f48beb7-e568-40fb-804e-f63444ccefe2/volumes/kubernetes.io~secret/v4-0-config-system-session tmpfs 15G 704K 15G 1% /var/lib/kubelet/pods/0f48beb7-e568-40fb-804e-f63444ccefe2/volumes/kubernetes.io~secret/v4-0-config-system-ocp-branding-template tmpfs 15G 0 15G 0% /var/lib/kubelet/pods/0f48beb7-e568-40fb-804e-f63444ccefe2/volumes/kubernetes.io~secret/v4-0-config-user-template-provider-selection tmpfs 15G 0 15G 0% /var/lib/kubelet/pods/0f48beb7-e568-40fb-804e-f63444ccefe2/volumes/kubernetes.io~secret/v4-0-config-user-template-login tmpfs 15G 256K 15G 1% /var/lib/kubelet/pods/0f48beb7-e568-40fb-804e-f63444ccefe2/volumes/kubernetes.io~projected/kube-api-access-sf5bj overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/82c4e02a04047982723a25e80845b1adcefb638fb0fca42f5bb35e53e0079150/merged shm 64M 0 64M 0% /run/containers/storage/overlay-containers/79e9d913207c9b2dc76dda1c0289120327eb6715be44a0cf53a2beb836852cf3/userdata/shm overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/94527efd6c1071a5fd63fb7f1d462f5a28a92f290052016c55ecac4b5aaaf05e/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/57ddadb12a13f7ab3df052bfaefa53e6d858aa55f8e4c25041d0dd96f8cd376b/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/6b02da4046d69676d31c2118a5d61c5f292e02a5a1196d5cf5058bf925c65aa0/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/b68714fd97c34cb91e890a31aa67f76cad6a17f6af51dbb0982282e251476092/merged overlay 120G 17G 104G 14% /var/lib/containers/storage/overlay/3c8ef5c5a69295092e1414743452d4e1318b63dcb78f1cab13db59765906da63/merged
Platform is ppc64le. OS info : on bastion : # cat /etc/redhat-release Red Hat Enterprise Linux release 8.4 (Ootpa) CoreOS nodes : [core@master-0 ~]$ cat /etc/redhat-release Red Hat Enterprise Linux CoreOS release 4.9
@alisha: What is the size of the disk that is being provisioned? Can you provide the output of `lsblk` and `df` from the control plane and worker nodes?
Taking this bug from the apiserver team as the multi-arch team thinks that the bug should be initially assigned to us. We will take a look at this bug and re-assign if necessary.
Master node : ============ # ssh core@master-0 lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 120G 0 disk ├─sda1 8:1 0 4M 0 part ├─sda3 8:3 0 384M 0 part └─sda4 8:4 0 119.6G 0 part /sysroot sdb 8:16 0 120G 0 disk ├─sdb1 8:17 0 4M 0 part ├─sdb3 8:19 0 384M 0 part /boot └─sdb4 8:20 0 119.6G 0 part # ssh core@master-0 df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 8267712 0 8267712 0% /dev tmpfs 8346304 256 8346048 1% /dev/shm tmpfs 8346304 453632 7892672 6% /run tmpfs 8346304 0 8346304 0% /sys/fs/cgroup /dev/sda4 125420524 11817044 113603480 10% /sysroot tmpfs 8346304 64 8346240 1% /tmp /dev/sdb3 372607 236374 112477 68% /boot overlay 8346304 453632 7892672 6% /etc/NetworkManager/systemConnectionsMerged tmpfs 1669248 0 1669248 0% /run/user/1000 Worker node : =========== # ssh core@worker-0 lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 120G 0 disk ├─sda1 8:1 0 4M 0 part ├─sda3 8:3 0 384M 0 part /boot └─sda4 8:4 0 119.6G 0 part /sysroot sdb 8:16 0 120G 0 disk ├─sdb1 8:17 0 4M 0 part ├─sdb3 8:19 0 384M 0 part └─sdb4 8:20 0 119.6G 0 part # ssh core@worker-0 df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 8267712 0 8267712 0% /dev tmpfs 8346304 128 8346176 1% /dev/shm tmpfs 8346304 246848 8099456 3% /run tmpfs 8346304 0 8346304 0% /sys/fs/cgroup /dev/sda4 125420524 8880312 116540212 8% /sysroot tmpfs 8346304 64 8346240 1% /tmp /dev/sda3 372607 236374 112477 68% /boot overlay 8346304 246848 8099456 3% /etc/NetworkManager/systemConnectionsMerged tmpfs 1669248 64 1669184 1% /run/user/1000 bastion : ======= # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 120G 0 disk ├─sda1 8:1 0 4M 0 part ├─sda2 8:2 0 120G 0 part └─mpatha 253:0 0 120G 0 mpath ├─mpatha1 253:2 0 4M 0 part └─mpatha2 253:3 0 120G 0 part / sdb 8:16 0 300G 0 disk └─mpathb 253:1 0 300G 0 mpath /export sdc 8:32 0 120G 0 disk ├─sdc1 8:33 0 4M 0 part ├─sdc2 8:34 0 120G 0 part └─mpatha 253:0 0 120G 0 mpath ├─mpatha1 253:2 0 4M 0 part └─mpatha2 253:3 0 120G 0 part / sdd 8:48 0 300G 0 disk └─mpathb 253:1 0 300G 0 mpath /export # df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 7795776 0 7795776 0% /dev tmpfs 7831936 192 7831744 1% /dev/shm tmpfs 7831936 789760 7042176 11% /run tmpfs 7831936 0 7831936 0% /sys/fs/cgroup /dev/mapper/mpatha2 125813740 11806620 114007120 10% / /dev/mapper/mpathb 308587328 894708 291947596 1% /export tmpfs 1566336 0 1566336 0% /run/user/0
From the oc output, it looks like this cluster was created 6 days ago and then there was some repeated failures which filled most of the /run tmpfs. This probably caused these cascading failures. Do we know when things started failing? was this cluster actively monitored to see if there were any issues? does the console have a history of alerts/events to report?
could a must-gather also be collected on this cluster?
can we login to the machine where we are hitting this out of space issue and see top 10 directories contributing to this, du -ha /run | sort -n -r | head -n 10
Output from a cluster with 4.9.0-0.nightly-ppc64le-2021-08-19-120135 Created 4 days ago. Which too has the same issue. # du -ha /run | sort -n -r | head -n 10 450M /run 448K /run/cloud-init 443M /run/log/journal 443M /run/log 427M /run/log/journal/8c700d6e302948dd8d2bda2d8db6bee4 320K /run/NetworkManager 192K /run/systemd/generator 192K /run/NetworkManager/devices 128K /run/systemd/transient 128K /run/systemd/sessions
must-gather exited with timeout : as shown below : # oc adm must-gather [must-gather ] OUT Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:945aa0a5d3d96df076bf139b19910683ac20fd63dd385821261c7317d25a3cb3 When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information. ClusterID: e9554f1e-e262-44b2-acf1-7274eb50a491 ClusterVersion: Stable at "4.9.0-0.nightly-ppc64le-2021-08-19-120135" ClusterOperators: clusteroperator/authentication is degraded because APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (container is waiting in apiserver-6774bf576b-8p6f9 pod) clusteroperator/console is not available (DeploymentAvailable: 1 replicas ready at version 4.9.0-0.nightly-ppc64le-2021-08-19-120135) because All is well clusteroperator/kube-apiserver is degraded because StaticPodsDegraded: pod/kube-apiserver-rdr-cicd-e6b7-mon01-master-0 container "kube-apiserver" is waiting: CreateContainerError: open /run/containers/storage/overlay-layers/.tmp-mountpoints.json607499636: no space left on device clusteroperator/kube-controller-manager is degraded because StaticPodsDegraded: pod/kube-controller-manager-rdr-cicd-e6b7-mon01-master-0 container "cluster-policy-controller" is waiting: CreateContainerError: open /run/containers/storage/overlay-layers/.tmp-mountpoints.json789204288: no space left on device StaticPodsDegraded: pod/kube-controller-manager-rdr-cicd-e6b7-mon01-master-0 container "kube-controller-manager" is waiting: CreateContainerError: open /run/containers/storage/overlay-layers/.tmp-mountpoints.json031946758: no space left on device StaticPodsDegraded: pod/kube-controller-manager-rdr-cicd-e6b7-mon01-master-1 container "cluster-policy-controller" is waiting: CreateContainerError: open /run/containers/storage/overlay-layers/.tmp-mountpoints.json127458881: no space left on device clusteroperator/kube-scheduler is degraded because StaticPodsDegraded: pod/openshift-kube-scheduler-rdr-cicd-e6b7-mon01-master-0 container "kube-scheduler" is waiting: CreateContainerError: open /run/containers/storage/overlay-layers/.tmp-mountpoints.json226094531: no space left on device StaticPodsDegraded: pod/openshift-kube-scheduler-rdr-cicd-e6b7-mon01-master-1 container "kube-scheduler" is waiting: CreateContainerError: open /run/containers/storage/overlay-layers/.tmp-mountpoints.json081094887: no space left on device clusteroperator/monitoring is not available (Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error.) because Failed to rollout the stack. Error: updating prometheus-adapter: reconciling PrometheusAdapter Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/prometheus-adapter: expected 3 replicas, got 2 updated replicas updating prometheus-k8s: waiting for Prometheus object changes failed: waiting for Prometheus openshift-monitoring/k8s: expected 2 replicas, got 1 updated replicas clusteroperator/network is degraded because DaemonSet "openshift-sdn/sdn" rollout is not making progress - last change 2021-08-25T03:05:51Z clusteroperator/service-ca is degraded because Degraded: the server was unable to return a response in the time allotted, but may still be processing the request (get configmaps signing-cabundle) [must-gather ] OUT namespace/openshift-must-gather-9qjbv created [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-bcltb created [must-gather ] OUT pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:945aa0a5d3d96df076bf139b19910683ac20fd63dd385821261c7317d25a3cb3 created [must-gather-rd462] OUT gather did not start: timed out waiting for the condition [must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-bcltb deleted [must-gather ] OUT namespace/openshift-must-gather-9qjbv deleted When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information. ClusterID: e9554f1e-e262-44b2-acf1-7274eb50a491 ClusterVersion: Stable at "4.9.0-0.nightly-ppc64le-2021-08-19-120135" ClusterOperators: clusteroperator/authentication is degraded because APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (container is waiting in apiserver-6774bf576b-8p6f9 pod) clusteroperator/console is not available (DeploymentAvailable: 1 replicas ready at version 4.9.0-0.nightly-ppc64le-2021-08-19-120135) because All is well clusteroperator/kube-apiserver is degraded because StaticPodsDegraded: pod/kube-apiserver-rdr-cicd-e6b7-mon01-master-0 container "kube-apiserver" is waiting: CreateContainerError: open /run/containers/storage/overlay-layers/.tmp-mountpoints.json841485797: no space left on device clusteroperator/kube-controller-manager is degraded because StaticPodsDegraded: pod/kube-controller-manager-rdr-cicd-e6b7-mon01-master-0 container "cluster-policy-controller" is waiting: CreateContainerError: open /run/containers/storage/overlay-layers/.tmp-mountpoints.json094090288: no space left on device StaticPodsDegraded: pod/kube-controller-manager-rdr-cicd-e6b7-mon01-master-0 container "kube-controller-manager" is waiting: CreateContainerError: open /run/containers/storage/overlay-layers/.tmp-mountpoints.json185948185: no space left on device StaticPodsDegraded: pod/kube-controller-manager-rdr-cicd-e6b7-mon01-master-1 container "cluster-policy-controller" is waiting: CreateContainerError: open /run/containers/storage/overlay-layers/.tmp-mountpoints.json080461409: no space left on device clusteroperator/kube-scheduler is degraded because StaticPodsDegraded: pod/openshift-kube-scheduler-rdr-cicd-e6b7-mon01-master-0 container "kube-scheduler" is waiting: CreateContainerError: open /run/containers/storage/overlay-layers/.tmp-mountpoints.json009285108: no space left on device StaticPodsDegraded: pod/openshift-kube-scheduler-rdr-cicd-e6b7-mon01-master-1 container "kube-scheduler" is waiting: CreateContainerError: open /run/containers/storage/overlay-layers/.tmp-mountpoints.json735593991: no space left on device clusteroperator/monitoring is not available (Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error.) because Failed to rollout the stack. Error: updating prometheus-adapter: reconciling PrometheusAdapter Deployment failed: updating Deployment object failed: waiting for DeploymentRollout of openshift-monitoring/prometheus-adapter: expected 3 replicas, got 2 updated replicas updating prometheus-k8s: waiting for Prometheus object changes failed: waiting for Prometheus openshift-monitoring/k8s: expected 2 replicas, got 1 updated replicas clusteroperator/network is degraded because DaemonSet "openshift-sdn/sdn" rollout is not making progress - last change 2021-08-25T03:05:51Z clusteroperator/service-ca is degraded because Degraded: the server was unable to return a response in the time allotted, but may still be processing the request (get configmaps signing-cabundle) Gathering data for ns/openshift-config... Gathering data for ns/openshift-config-managed... Gathering data for ns/openshift-authentication... Gathering data for ns/openshift-authentication-operator... Gathering data for ns/openshift-ingress... Gathering data for ns/openshift-oauth-apiserver... Gathering data for ns/openshift-machine-api... Gathering data for ns/openshift-cloud-controller-manager-operator... Gathering data for ns/openshift-cloud-controller-manager... Gathering data for ns/openshift-cloud-credential-operator... Gathering data for ns/openshift-config-operator... Gathering data for ns/openshift-console-operator... Gathering data for ns/openshift-console... Gathering data for ns/openshift-cluster-storage-operator... Gathering data for ns/openshift-dns-operator... Gathering data for ns/openshift-dns... Gathering data for ns/openshift-etcd-operator... Gathering data for ns/openshift-etcd... Gathering data for ns/openshift-image-registry... Gathering data for ns/openshift-ingress-operator... Gathering data for ns/openshift-ingress-canary... Gathering data for ns/openshift-insights... Gathering data for ns/openshift-kube-apiserver-operator... Gathering data for ns/openshift-kube-apiserver... Gathering data for ns/openshift-kube-controller-manager... Gathering data for ns/openshift-kube-controller-manager-operator... Gathering data for ns/openshift-kube-scheduler... Gathering data for ns/openshift-kube-scheduler-operator... Gathering data for ns/openshift-kube-storage-version-migrator... Gathering data for ns/openshift-kube-storage-version-migrator-operator... Gathering data for ns/openshift-cluster-machine-approver... Gathering data for ns/openshift-machine-config-operator... Gathering data for ns/openshift-kni-infra... Gathering data for ns/openshift-openstack-infra... Gathering data for ns/openshift-ovirt-infra... Gathering data for ns/openshift-vsphere-infra... Gathering data for ns/openshift-marketplace... Gathering data for ns/openshift-monitoring... Gathering data for ns/openshift-user-workload-monitoring... Gathering data for ns/openshift-multus... Gathering data for ns/openshift-sdn... Gathering data for ns/openshift-host-network... Gathering data for ns/openshift-network-diagnostics... Gathering data for ns/openshift-network-operator... Gathering data for ns/openshift-cluster-node-tuning-operator... Gathering data for ns/openshift-apiserver-operator... Gathering data for ns/openshift-apiserver... Gathering data for ns/openshift-controller-manager-operator... Gathering data for ns/openshift-controller-manager... Gathering data for ns/openshift-cluster-samples-operator... Gathering data for ns/openshift... Gathering data for ns/openshift-operator-lifecycle-manager... Gathering data for ns/openshift-service-ca-operator... Gathering data for ns/openshift-service-ca... Gathering data for ns/openshift-cluster-csi-drivers... Wrote inspect data to must-gather.local.4336199250967024295. error running backup collection: errors ocurred while gathering data: [unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request, skipping gathering namespaces/openshift-authentication due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-authentication one or more errors ocurred while gathering container data for pod oauth-openshift-7d57f74c66-tb4bk: [container "oauth-openshift" in pod "oauth-openshift-7d57f74c66-tb4bk" is waiting to start: ContainerCreating, previous terminated container "oauth-openshift" in pod "oauth-openshift-7d57f74c66-tb4bk" not found], skipping gathering namespaces/openshift-machine-api due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-machine-api [one or more errors ocurred while gathering container data for pod cluster-baremetal-operator-749d46f4d9-k6pzq: [container "cluster-baremetal-operator" in pod "cluster-baremetal-operator-749d46f4d9-k6pzq" is waiting to start: ContainerCreating, previous terminated container "cluster-baremetal-operator" in pod "cluster-baremetal-operator-749d46f4d9-k6pzq" not found, previous terminated container "kube-rbac-proxy" in pod "cluster-baremetal-operator-749d46f4d9-k6pzq" not found, container "kube-rbac-proxy" in pod "cluster-baremetal-operator-749d46f4d9-k6pzq" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod machine-api-operator-7b88cf8d4-cljtt: [container "kube-rbac-proxy" in pod "machine-api-operator-7b88cf8d4-cljtt" is waiting to start: ContainerCreating, previous terminated container "kube-rbac-proxy" in pod "machine-api-operator-7b88cf8d4-cljtt" not found, container "machine-api-operator" in pod "machine-api-operator-7b88cf8d4-cljtt" is waiting to start: ContainerCreating, previous terminated container "machine-api-operator" in pod "machine-api-operator-7b88cf8d4-cljtt" not found]], skipping gathering namespaces/openshift-cloud-credential-operator due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-cloud-credential-operator one or more errors ocurred while gathering container data for pod cloud-credential-operator-f46d67f5b-g2khs: [previous terminated container "kube-rbac-proxy" in pod "cloud-credential-operator-f46d67f5b-g2khs" not found, container "kube-rbac-proxy" in pod "cloud-credential-operator-f46d67f5b-g2khs" is waiting to start: ContainerCreating, previous terminated container "cloud-credential-operator" in pod "cloud-credential-operator-f46d67f5b-g2khs" not found, container "cloud-credential-operator" in pod "cloud-credential-operator-f46d67f5b-g2khs" is waiting to start: ContainerCreating], skipping gathering namespaces/openshift-config-operator due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-config-operator one or more errors ocurred while gathering container data for pod openshift-config-operator-748d9d4d57-hjp4v: [previous terminated container "openshift-config-operator" in pod "openshift-config-operator-748d9d4d57-hjp4v" not found, container "openshift-config-operator" in pod "openshift-config-operator-748d9d4d57-hjp4v" is waiting to start: ContainerCreating], skipping gathering namespaces/openshift-console due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-console [one or more errors ocurred while gathering container data for pod console-85b77c55f5-zfl6h: [previous terminated container "console" in pod "console-85b77c55f5-zfl6h" not found, container "console" in pod "console-85b77c55f5-zfl6h" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod downloads-7899d55494-6n2kl: [container "download-server" in pod "downloads-7899d55494-6n2kl" is waiting to start: ContainerCreating, previous terminated container "download-server" in pod "downloads-7899d55494-6n2kl" not found]], skipping gathering namespaces/openshift-cluster-storage-operator due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-cluster-storage-operator [one or more errors ocurred while gathering container data for pod csi-snapshot-controller-74cbcb446f-q7bff: [previous terminated container "snapshot-controller" in pod "csi-snapshot-controller-74cbcb446f-q7bff" not found, container "snapshot-controller" in pod "csi-snapshot-controller-74cbcb446f-q7bff" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod csi-snapshot-controller-operator-6b594f57bf-85qt7: [container "csi-snapshot-controller-operator" in pod "csi-snapshot-controller-operator-6b594f57bf-85qt7" is waiting to start: ContainerCreating, previous terminated container "csi-snapshot-controller-operator" in pod "csi-snapshot-controller-operator-6b594f57bf-85qt7" not found], one or more errors ocurred while gathering container data for pod csi-snapshot-webhook-6565589b6f-w2m5n: [previous terminated container "webhook" in pod "csi-snapshot-webhook-6565589b6f-w2m5n" not found, container "webhook" in pod "csi-snapshot-webhook-6565589b6f-w2m5n" is waiting to start: ContainerCreating]], skipping gathering namespaces/openshift-dns-operator due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-dns-operator one or more errors ocurred while gathering container data for pod dns-operator-5888f98bfd-gltf8: [previous terminated container "dns-operator" in pod "dns-operator-5888f98bfd-gltf8" not found, container "dns-operator" in pod "dns-operator-5888f98bfd-gltf8" is waiting to start: ContainerCreating, container "kube-rbac-proxy" in pod "dns-operator-5888f98bfd-gltf8" is waiting to start: ContainerCreating, previous terminated container "kube-rbac-proxy" in pod "dns-operator-5888f98bfd-gltf8" not found], skipping gathering namespaces/openshift-etcd-operator due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-etcd-operator one or more errors ocurred while gathering container data for pod etcd-operator-7bdb4b7779-rdsk9: [previous terminated container "etcd-operator" in pod "etcd-operator-7bdb4b7779-rdsk9" not found, container "etcd-operator" in pod "etcd-operator-7bdb4b7779-rdsk9" is waiting to start: ContainerCreating], skipping gathering namespaces/openshift-image-registry due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-image-registry one or more errors ocurred while gathering container data for pod image-pruner-27164160--1-2777n: [container "image-pruner" in pod "image-pruner-27164160--1-2777n" is waiting to start: ContainerCreating, previous terminated container "image-pruner" in pod "image-pruner-27164160--1-2777n" not found], skipping gathering namespaces/openshift-ingress-operator due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-ingress-operator one or more errors ocurred while gathering container data for pod ingress-operator-7448c475d8-p4ssm: [container "ingress-operator" in pod "ingress-operator-7448c475d8-p4ssm" is waiting to start: ContainerCreating, previous terminated container "ingress-operator" in pod "ingress-operator-7448c475d8-p4ssm" not found, container "kube-rbac-proxy" in pod "ingress-operator-7448c475d8-p4ssm" is waiting to start: ContainerCreating, previous terminated container "kube-rbac-proxy" in pod "ingress-operator-7448c475d8-p4ssm" not found], skipping gathering secrets/support due to error: secrets "support" not found, skipping gathering namespaces/openshift-cluster-machine-approver due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-cluster-machine-approver one or more errors ocurred while gathering container data for pod machine-approver-697778c65-v4d56: [previous terminated container "kube-rbac-proxy" in pod "machine-approver-697778c65-v4d56" not found, container "kube-rbac-proxy" in pod "machine-approver-697778c65-v4d56" is waiting to start: ContainerCreating, container "machine-approver-controller" in pod "machine-approver-697778c65-v4d56" is waiting to start: ContainerCreating, previous terminated container "machine-approver-controller" in pod "machine-approver-697778c65-v4d56" not found], skipping gathering namespaces/openshift-machine-config-operator due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-machine-config-operator [one or more errors ocurred while gathering container data for pod machine-config-controller-56449d7c78-9f4xp: [previous terminated container "machine-config-controller" in pod "machine-config-controller-56449d7c78-9f4xp" not found, container "machine-config-controller" in pod "machine-config-controller-56449d7c78-9f4xp" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod machine-config-operator-69977ff6f9-n7csm: [previous terminated container "machine-config-operator" in pod "machine-config-operator-69977ff6f9-n7csm" not found, container "machine-config-operator" in pod "machine-config-operator-69977ff6f9-n7csm" is waiting to start: ContainerCreating]], skipping gathering namespaces/openshift-marketplace due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-marketplace [one or more errors ocurred while gathering container data for pod certified-operators-g9qkv: [container "registry-server" in pod "certified-operators-g9qkv" is waiting to start: ImageInspectError, previous terminated container "registry-server" in pod "certified-operators-g9qkv" not found], one or more errors ocurred while gathering container data for pod community-operators-fn2f4: [container "registry-server" in pod "community-operators-fn2f4" is waiting to start: ImageInspectError, previous terminated container "registry-server" in pod "community-operators-fn2f4" not found], one or more errors ocurred while gathering container data for pod redhat-marketplace-gsrdp: [previous terminated container "registry-server" in pod "redhat-marketplace-gsrdp" not found, container "registry-server" in pod "redhat-marketplace-gsrdp" is waiting to start: ImageInspectError], one or more errors ocurred while gathering container data for pod redhat-operators-6jxrl: [previous terminated container "registry-server" in pod "redhat-operators-6jxrl" not found, container "registry-server" in pod "redhat-operators-6jxrl" is waiting to start: ImageInspectError]], skipping gathering namespaces/openshift-monitoring due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-monitoring [one or more errors ocurred while gathering container data for pod cluster-monitoring-operator-7cbbf987c4-gxdbp: [previous terminated container "kube-rbac-proxy" in pod "cluster-monitoring-operator-7cbbf987c4-gxdbp" not found, container "kube-rbac-proxy" in pod "cluster-monitoring-operator-7cbbf987c4-gxdbp" is waiting to start: ContainerCreating, previous terminated container "cluster-monitoring-operator" in pod "cluster-monitoring-operator-7cbbf987c4-gxdbp" not found, container "cluster-monitoring-operator" in pod "cluster-monitoring-operator-7cbbf987c4-gxdbp" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod prometheus-adapter-6d9c5fd87-nlcnb: [previous terminated container "prometheus-adapter" in pod "prometheus-adapter-6d9c5fd87-nlcnb" not found, container "prometheus-adapter" in pod "prometheus-adapter-6d9c5fd87-nlcnb" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod prometheus-k8s-0: [container "prometheus" in pod "prometheus-k8s-0" is waiting to start: PodInitializing, previous terminated container "prometheus" in pod "prometheus-k8s-0" not found, container "config-reloader" in pod "prometheus-k8s-0" is waiting to start: PodInitializing, previous terminated container "config-reloader" in pod "prometheus-k8s-0" not found, container "thanos-sidecar" in pod "prometheus-k8s-0" is waiting to start: PodInitializing, previous terminated container "thanos-sidecar" in pod "prometheus-k8s-0" not found, previous terminated container "prometheus-proxy" in pod "prometheus-k8s-0" not found, container "prometheus-proxy" in pod "prometheus-k8s-0" is waiting to start: PodInitializing, container "kube-rbac-proxy" in pod "prometheus-k8s-0" is waiting to start: PodInitializing, previous terminated container "kube-rbac-proxy" in pod "prometheus-k8s-0" not found, previous terminated container "prom-label-proxy" in pod "prometheus-k8s-0" not found, container "prom-label-proxy" in pod "prometheus-k8s-0" is waiting to start: PodInitializing, container "kube-rbac-proxy-thanos" in pod "prometheus-k8s-0" is waiting to start: PodInitializing, previous terminated container "kube-rbac-proxy-thanos" in pod "prometheus-k8s-0" not found, previous terminated container "init-config-reloader" in pod "prometheus-k8s-0" not found, container "init-config-reloader" in pod "prometheus-k8s-0" is waiting to start: PodInitializing]], skipping gathering EgressFirewall.k8s.ovn.org due to error: the server doesn't have a resource type "EgressFirewall", skipping gathering EgressIP.k8s.ovn.org due to error: the server doesn't have a resource type "EgressIP", skipping gathering endpoints/host-etcd-2 due to error: endpoints "host-etcd-2" not found, skipping gathering namespaces/openshift-cluster-samples-operator due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-cluster-samples-operator one or more errors ocurred while gathering container data for pod cluster-samples-operator-664497f4d8-dgx4d: [previous terminated container "cluster-samples-operator" in pod "cluster-samples-operator-664497f4d8-dgx4d" not found, container "cluster-samples-operator" in pod "cluster-samples-operator-664497f4d8-dgx4d" is waiting to start: ContainerCreating, container "cluster-samples-operator-watch" in pod "cluster-samples-operator-664497f4d8-dgx4d" is waiting to start: ContainerCreating, previous terminated container "cluster-samples-operator-watch" in pod "cluster-samples-operator-664497f4d8-dgx4d" not found], skipping gathering namespaces/openshift-operator-lifecycle-manager due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-operator-lifecycle-manager [one or more errors ocurred while gathering container data for pod collect-profiles-27159675--1-jjxs2: [previous terminated container "collect-profiles" in pod "collect-profiles-27159675--1-jjxs2" not found, container "collect-profiles" in pod "collect-profiles-27159675--1-jjxs2" is waiting to start: CreateContainerError], one or more errors ocurred while gathering container data for pod collect-profiles-27159840--1-rjqd4: [previous terminated container "collect-profiles" in pod "collect-profiles-27159840--1-rjqd4" not found, container "collect-profiles" in pod "collect-profiles-27159840--1-rjqd4" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27161040--1-sj9z8: [container "collect-profiles" in pod "collect-profiles-27161040--1-sj9z8" is waiting to start: CreateContainerError, previous terminated container "collect-profiles" in pod "collect-profiles-27161040--1-sj9z8" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27161265--1-znp5s: [previous terminated container "collect-profiles" in pod "collect-profiles-27161265--1-znp5s" not found, container "collect-profiles" in pod "collect-profiles-27161265--1-znp5s" is waiting to start: CreateContainerError], one or more errors ocurred while gathering container data for pod collect-profiles-27161310--1-wd5vb: [previous terminated container "collect-profiles" in pod "collect-profiles-27161310--1-wd5vb" not found, container "collect-profiles" in pod "collect-profiles-27161310--1-wd5vb" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27161460--1-q7gfb: [container "collect-profiles" in pod "collect-profiles-27161460--1-q7gfb" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27161460--1-q7gfb" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27161490--1-jwwm5: [container "collect-profiles" in pod "collect-profiles-27161490--1-jwwm5" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27161490--1-jwwm5" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163125--1-2bh7c: [container "collect-profiles" in pod "collect-profiles-27163125--1-2bh7c" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163125--1-2bh7c" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163140--1-vvj6l: [previous terminated container "collect-profiles" in pod "collect-profiles-27163140--1-vvj6l" not found, container "collect-profiles" in pod "collect-profiles-27163140--1-vvj6l" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163155--1-n22lp: [container "collect-profiles" in pod "collect-profiles-27163155--1-n22lp" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163155--1-n22lp" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163170--1-9w5sl: [previous terminated container "collect-profiles" in pod "collect-profiles-27163170--1-9w5sl" not found, container "collect-profiles" in pod "collect-profiles-27163170--1-9w5sl" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163185--1-m6gr5: [container "collect-profiles" in pod "collect-profiles-27163185--1-m6gr5" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163185--1-m6gr5" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163200--1-9jpk8: [container "collect-profiles" in pod "collect-profiles-27163200--1-9jpk8" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163200--1-9jpk8" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163215--1-d2lrd: [previous terminated container "collect-profiles" in pod "collect-profiles-27163215--1-d2lrd" not found, container "collect-profiles" in pod "collect-profiles-27163215--1-d2lrd" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163230--1-bj2vg: [container "collect-profiles" in pod "collect-profiles-27163230--1-bj2vg" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163230--1-bj2vg" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163245--1-wqlrl: [container "collect-profiles" in pod "collect-profiles-27163245--1-wqlrl" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163245--1-wqlrl" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163260--1-jjnps: [previous terminated container "collect-profiles" in pod "collect-profiles-27163260--1-jjnps" not found, container "collect-profiles" in pod "collect-profiles-27163260--1-jjnps" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163275--1-2xzsm: [previous terminated container "collect-profiles" in pod "collect-profiles-27163275--1-2xzsm" not found, container "collect-profiles" in pod "collect-profiles-27163275--1-2xzsm" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163290--1-hssds: [previous terminated container "collect-profiles" in pod "collect-profiles-27163290--1-hssds" not found, container "collect-profiles" in pod "collect-profiles-27163290--1-hssds" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163305--1-9s8qc: [container "collect-profiles" in pod "collect-profiles-27163305--1-9s8qc" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163305--1-9s8qc" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163320--1-c86v5: [previous terminated container "collect-profiles" in pod "collect-profiles-27163320--1-c86v5" not found, container "collect-profiles" in pod "collect-profiles-27163320--1-c86v5" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163335--1-ssrpk: [container "collect-profiles" in pod "collect-profiles-27163335--1-ssrpk" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163335--1-ssrpk" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163350--1-xstrt: [previous terminated container "collect-profiles" in pod "collect-profiles-27163350--1-xstrt" not found, container "collect-profiles" in pod "collect-profiles-27163350--1-xstrt" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163365--1-q468s: [container "collect-profiles" in pod "collect-profiles-27163365--1-q468s" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163365--1-q468s" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163380--1-xsht6: [previous terminated container "collect-profiles" in pod "collect-profiles-27163380--1-xsht6" not found, container "collect-profiles" in pod "collect-profiles-27163380--1-xsht6" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163395--1-njd2g: [previous terminated container "collect-profiles" in pod "collect-profiles-27163395--1-njd2g" not found, container "collect-profiles" in pod "collect-profiles-27163395--1-njd2g" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163410--1-k9gqd: [previous terminated container "collect-profiles" in pod "collect-profiles-27163410--1-k9gqd" not found, container "collect-profiles" in pod "collect-profiles-27163410--1-k9gqd" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163425--1-x7d9p: [container "collect-profiles" in pod "collect-profiles-27163425--1-x7d9p" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163425--1-x7d9p" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163440--1-wpphx: [previous terminated container "collect-profiles" in pod "collect-profiles-27163440--1-wpphx" not found, container "collect-profiles" in pod "collect-profiles-27163440--1-wpphx" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163455--1-qpjrd: [container "collect-profiles" in pod "collect-profiles-27163455--1-qpjrd" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163455--1-qpjrd" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163470--1-gftz4: [previous terminated container "collect-profiles" in pod "collect-profiles-27163470--1-gftz4" not found, container "collect-profiles" in pod "collect-profiles-27163470--1-gftz4" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163485--1-46dbd: [previous terminated container "collect-profiles" in pod "collect-profiles-27163485--1-46dbd" not found, container "collect-profiles" in pod "collect-profiles-27163485--1-46dbd" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163500--1-c4r7q: [previous terminated container "collect-profiles" in pod "collect-profiles-27163500--1-c4r7q" not found, container "collect-profiles" in pod "collect-profiles-27163500--1-c4r7q" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163515--1-tgzvc: [container "collect-profiles" in pod "collect-profiles-27163515--1-tgzvc" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163515--1-tgzvc" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163530--1-p6rqv: [container "collect-profiles" in pod "collect-profiles-27163530--1-p6rqv" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163530--1-p6rqv" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163545--1-gv4k8: [container "collect-profiles" in pod "collect-profiles-27163545--1-gv4k8" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163545--1-gv4k8" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163560--1-j7647: [previous terminated container "collect-profiles" in pod "collect-profiles-27163560--1-j7647" not found, container "collect-profiles" in pod "collect-profiles-27163560--1-j7647" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163575--1-49kqt: [container "collect-profiles" in pod "collect-profiles-27163575--1-49kqt" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163575--1-49kqt" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163590--1-gz22k: [previous terminated container "collect-profiles" in pod "collect-profiles-27163590--1-gz22k" not found, container "collect-profiles" in pod "collect-profiles-27163590--1-gz22k" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163605--1-swvbl: [container "collect-profiles" in pod "collect-profiles-27163605--1-swvbl" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163605--1-swvbl" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163620--1-kvbjw: [previous terminated container "collect-profiles" in pod "collect-profiles-27163620--1-kvbjw" not found, container "collect-profiles" in pod "collect-profiles-27163620--1-kvbjw" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163635--1-gcjzm: [container "collect-profiles" in pod "collect-profiles-27163635--1-gcjzm" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163635--1-gcjzm" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163650--1-hngp8: [container "collect-profiles" in pod "collect-profiles-27163650--1-hngp8" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163650--1-hngp8" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163665--1-l7jgm: [previous terminated container "collect-profiles" in pod "collect-profiles-27163665--1-l7jgm" not found, container "collect-profiles" in pod "collect-profiles-27163665--1-l7jgm" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163680--1-nq44n: [container "collect-profiles" in pod "collect-profiles-27163680--1-nq44n" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163680--1-nq44n" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163695--1-r7phm: [container "collect-profiles" in pod "collect-profiles-27163695--1-r7phm" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163695--1-r7phm" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163710--1-zg66z: [previous terminated container "collect-profiles" in pod "collect-profiles-27163710--1-zg66z" not found, container "collect-profiles" in pod "collect-profiles-27163710--1-zg66z" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163725--1-j8wm2: [previous terminated container "collect-profiles" in pod "collect-profiles-27163725--1-j8wm2" not found, container "collect-profiles" in pod "collect-profiles-27163725--1-j8wm2" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163740--1-r7pmr: [container "collect-profiles" in pod "collect-profiles-27163740--1-r7pmr" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163740--1-r7pmr" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163755--1-tbnzc: [previous terminated container "collect-profiles" in pod "collect-profiles-27163755--1-tbnzc" not found, container "collect-profiles" in pod "collect-profiles-27163755--1-tbnzc" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163770--1-kh6ct: [previous terminated container "collect-profiles" in pod "collect-profiles-27163770--1-kh6ct" not found, container "collect-profiles" in pod "collect-profiles-27163770--1-kh6ct" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163785--1-bl7r2: [previous terminated container "collect-profiles" in pod "collect-profiles-27163785--1-bl7r2" not found, container "collect-profiles" in pod "collect-profiles-27163785--1-bl7r2" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163800--1-gd5sg: [container "collect-profiles" in pod "collect-profiles-27163800--1-gd5sg" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163800--1-gd5sg" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163815--1-glf4d: [container "collect-profiles" in pod "collect-profiles-27163815--1-glf4d" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163815--1-glf4d" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163830--1-ds752: [container "collect-profiles" in pod "collect-profiles-27163830--1-ds752" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163830--1-ds752" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163845--1-q5ccz: [container "collect-profiles" in pod "collect-profiles-27163845--1-q5ccz" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163845--1-q5ccz" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163860--1-78t27: [previous terminated container "collect-profiles" in pod "collect-profiles-27163860--1-78t27" not found, container "collect-profiles" in pod "collect-profiles-27163860--1-78t27" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163875--1-7x9p9: [previous terminated container "collect-profiles" in pod "collect-profiles-27163875--1-7x9p9" not found, container "collect-profiles" in pod "collect-profiles-27163875--1-7x9p9" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163890--1-xhgfp: [container "collect-profiles" in pod "collect-profiles-27163890--1-xhgfp" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163890--1-xhgfp" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163905--1-x9zjv: [previous terminated container "collect-profiles" in pod "collect-profiles-27163905--1-x9zjv" not found, container "collect-profiles" in pod "collect-profiles-27163905--1-x9zjv" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163920--1-v688l: [previous terminated container "collect-profiles" in pod "collect-profiles-27163920--1-v688l" not found, container "collect-profiles" in pod "collect-profiles-27163920--1-v688l" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163935--1-cxd5l: [previous terminated container "collect-profiles" in pod "collect-profiles-27163935--1-cxd5l" not found, container "collect-profiles" in pod "collect-profiles-27163935--1-cxd5l" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163950--1-ww2s8: [previous terminated container "collect-profiles" in pod "collect-profiles-27163950--1-ww2s8" not found, container "collect-profiles" in pod "collect-profiles-27163950--1-ww2s8" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163965--1-zzx6d: [previous terminated container "collect-profiles" in pod "collect-profiles-27163965--1-zzx6d" not found, container "collect-profiles" in pod "collect-profiles-27163965--1-zzx6d" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27163980--1-cp7wf: [container "collect-profiles" in pod "collect-profiles-27163980--1-cp7wf" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163980--1-cp7wf" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27163995--1-5flwd: [container "collect-profiles" in pod "collect-profiles-27163995--1-5flwd" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27163995--1-5flwd" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164010--1-g7l9f: [container "collect-profiles" in pod "collect-profiles-27164010--1-g7l9f" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164010--1-g7l9f" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164025--1-lhzrl: [container "collect-profiles" in pod "collect-profiles-27164025--1-lhzrl" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164025--1-lhzrl" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164040--1-qsbmh: [container "collect-profiles" in pod "collect-profiles-27164040--1-qsbmh" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164040--1-qsbmh" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164055--1-wlmcg: [container "collect-profiles" in pod "collect-profiles-27164055--1-wlmcg" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164055--1-wlmcg" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164070--1-2nqf6: [previous terminated container "collect-profiles" in pod "collect-profiles-27164070--1-2nqf6" not found, container "collect-profiles" in pod "collect-profiles-27164070--1-2nqf6" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164085--1-j8sld: [container "collect-profiles" in pod "collect-profiles-27164085--1-j8sld" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164085--1-j8sld" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164100--1-9f4rz: [container "collect-profiles" in pod "collect-profiles-27164100--1-9f4rz" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164100--1-9f4rz" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164115--1-5tptl: [container "collect-profiles" in pod "collect-profiles-27164115--1-5tptl" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164115--1-5tptl" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164130--1-8tzqj: [container "collect-profiles" in pod "collect-profiles-27164130--1-8tzqj" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164130--1-8tzqj" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164145--1-z6v5h: [container "collect-profiles" in pod "collect-profiles-27164145--1-z6v5h" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164145--1-z6v5h" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164160--1-knq2g: [previous terminated container "collect-profiles" in pod "collect-profiles-27164160--1-knq2g" not found, container "collect-profiles" in pod "collect-profiles-27164160--1-knq2g" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164175--1-4wq2v: [previous terminated container "collect-profiles" in pod "collect-profiles-27164175--1-4wq2v" not found, container "collect-profiles" in pod "collect-profiles-27164175--1-4wq2v" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164190--1-w4f2x: [container "collect-profiles" in pod "collect-profiles-27164190--1-w4f2x" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164190--1-w4f2x" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164205--1-zxhs7: [previous terminated container "collect-profiles" in pod "collect-profiles-27164205--1-zxhs7" not found, container "collect-profiles" in pod "collect-profiles-27164205--1-zxhs7" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164220--1-hf56p: [container "collect-profiles" in pod "collect-profiles-27164220--1-hf56p" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164220--1-hf56p" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164235--1-pqvsd: [previous terminated container "collect-profiles" in pod "collect-profiles-27164235--1-pqvsd" not found, container "collect-profiles" in pod "collect-profiles-27164235--1-pqvsd" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164250--1-h6w56: [container "collect-profiles" in pod "collect-profiles-27164250--1-h6w56" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164250--1-h6w56" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164265--1-q9lzk: [previous terminated container "collect-profiles" in pod "collect-profiles-27164265--1-q9lzk" not found, container "collect-profiles" in pod "collect-profiles-27164265--1-q9lzk" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164280--1-fwwr4: [previous terminated container "collect-profiles" in pod "collect-profiles-27164280--1-fwwr4" not found, container "collect-profiles" in pod "collect-profiles-27164280--1-fwwr4" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164295--1-zlrvb: [container "collect-profiles" in pod "collect-profiles-27164295--1-zlrvb" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164295--1-zlrvb" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164310--1-hgjjj: [previous terminated container "collect-profiles" in pod "collect-profiles-27164310--1-hgjjj" not found, container "collect-profiles" in pod "collect-profiles-27164310--1-hgjjj" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164325--1-wgxsf: [previous terminated container "collect-profiles" in pod "collect-profiles-27164325--1-wgxsf" not found, container "collect-profiles" in pod "collect-profiles-27164325--1-wgxsf" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164340--1-2slj5: [container "collect-profiles" in pod "collect-profiles-27164340--1-2slj5" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164340--1-2slj5" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164355--1-62nvd: [container "collect-profiles" in pod "collect-profiles-27164355--1-62nvd" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164355--1-62nvd" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164370--1-kcnsg: [previous terminated container "collect-profiles" in pod "collect-profiles-27164370--1-kcnsg" not found, container "collect-profiles" in pod "collect-profiles-27164370--1-kcnsg" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164385--1-fndc6: [container "collect-profiles" in pod "collect-profiles-27164385--1-fndc6" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164385--1-fndc6" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164400--1-2nzmb: [container "collect-profiles" in pod "collect-profiles-27164400--1-2nzmb" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164400--1-2nzmb" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164415--1-ptb8r: [container "collect-profiles" in pod "collect-profiles-27164415--1-ptb8r" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164415--1-ptb8r" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164430--1-xjlh2: [container "collect-profiles" in pod "collect-profiles-27164430--1-xjlh2" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164430--1-xjlh2" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164445--1-svfq4: [previous terminated container "collect-profiles" in pod "collect-profiles-27164445--1-svfq4" not found, container "collect-profiles" in pod "collect-profiles-27164445--1-svfq4" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164460--1-qpzmf: [previous terminated container "collect-profiles" in pod "collect-profiles-27164460--1-qpzmf" not found, container "collect-profiles" in pod "collect-profiles-27164460--1-qpzmf" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164475--1-4sxsk: [previous terminated container "collect-profiles" in pod "collect-profiles-27164475--1-4sxsk" not found, container "collect-profiles" in pod "collect-profiles-27164475--1-4sxsk" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164490--1-qsc49: [container "collect-profiles" in pod "collect-profiles-27164490--1-qsc49" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164490--1-qsc49" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164505--1-qfc6r: [previous terminated container "collect-profiles" in pod "collect-profiles-27164505--1-qfc6r" not found, container "collect-profiles" in pod "collect-profiles-27164505--1-qfc6r" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164520--1-xpbq5: [previous terminated container "collect-profiles" in pod "collect-profiles-27164520--1-xpbq5" not found, container "collect-profiles" in pod "collect-profiles-27164520--1-xpbq5" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164535--1-z7k4n: [previous terminated container "collect-profiles" in pod "collect-profiles-27164535--1-z7k4n" not found, container "collect-profiles" in pod "collect-profiles-27164535--1-z7k4n" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164550--1-9rq2x: [previous terminated container "collect-profiles" in pod "collect-profiles-27164550--1-9rq2x" not found, container "collect-profiles" in pod "collect-profiles-27164550--1-9rq2x" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164565--1-shv6q: [container "collect-profiles" in pod "collect-profiles-27164565--1-shv6q" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164565--1-shv6q" not found], one or more errors ocurred while gathering container data for pod collect-profiles-27164580--1-gxcng: [previous terminated container "collect-profiles" in pod "collect-profiles-27164580--1-gxcng" not found, container "collect-profiles" in pod "collect-profiles-27164580--1-gxcng" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164595--1-zsc2z: [previous terminated container "collect-profiles" in pod "collect-profiles-27164595--1-zsc2z" not found, container "collect-profiles" in pod "collect-profiles-27164595--1-zsc2z" is waiting to start: ContainerCreating], one or more errors ocurred while gathering container data for pod collect-profiles-27164610--1-grr9b: [container "collect-profiles" in pod "collect-profiles-27164610--1-grr9b" is waiting to start: ContainerCreating, previous terminated container "collect-profiles" in pod "collect-profiles-27164610--1-grr9b" not found], one or more errors ocurred while gathering container data for pod package-server-manager-77f8d8b554-j5g6j: [container "package-server-manager" in pod "package-server-manager-77f8d8b554-j5g6j" is waiting to start: ContainerCreating, previous terminated container "package-server-manager" in pod "package-server-manager-77f8d8b554-j5g6j" not found]], skipping gathering namespaces/openshift-service-ca-operator due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-service-ca-operator one or more errors ocurred while gathering container data for pod service-ca-operator-7667c9f4df-cmfhg: [container "service-ca-operator" in pod "service-ca-operator-7667c9f4df-cmfhg" is waiting to start: ContainerCreating, previous terminated container "service-ca-operator" in pod "service-ca-operator-7667c9f4df-cmfhg" not found], skipping gathering namespaces/openshift-service-ca due to error: one or more errors ocurred while gathering pod-specific data for namespace: openshift-service-ca one or more errors ocurred while gathering container data for pod service-ca-997679556-dbr8j: [previous terminated container "service-ca-controller" in pod "service-ca-997679556-dbr8j" not found, container "service-ca-controller" in pod "service-ca-997679556-dbr8j" is waiting to start: ContainerCreating], skipping gathering namespaces/openshift-manila-csi-driver due to error: namespaces "openshift-manila-csi-driver" not found]error: gather did not start for pod must-gather-rd462: timed out waiting for the condition
oh actually what is filling up is tmpfs (/run) which comes from RAM. what is the memory usage of the system ? can we understand if there is sufficient memory on the system.
but that being said, looking at the original comment /run is a tmpfs of 8G and almost 100% of it is used up. can we get more details on what is causing /run to fill up? what are the files/folders you see there? For example from the previous comment on the contents of /run: 427M /run/log/journal/8c700d6e302948dd8d2bda2d8db6bee4 can we get some snippet of this journal log to see what is causing it to be 427M ? i believe if we can find that maybe we can find the underlying cause. Also that output only shows /run being around 450M whereas in the original comment it almost filled up 8G. can we please get the output when /run is actually filled up ?
What was filling up /run was a bunch of files in /run/crio/exec-pid-dir. Each file is 64k in size, even though each has 6-7 bytes with the PID. When the number of files get to 125k, the 8G file-system fills up. The number of files in /run/crio/exec-pid-dir is growing at a rate of ~34/minute, which means that an idle cluster in about 2+ days would fill up /run. [root@rdr-cicd-e6b7-mon01-master-1 exec-pid-dir]# while true; do ls |wc; sleep 60; done 7998 7998 807798 8034 8034 811434 8070 8070 815070 The processes seem to be related to a collect-profiles cronjob. [root@rdr-cicd-e6b7-mon01-bastion-0 ~]# oc get all NAME READY STATUS RESTARTS AGE pod/catalog-operator-7d866fcb7b-7748m 1/1 Running 38 5d pod/collect-profiles-27161040--1-sj9z8 0/1 Terminating 0 5d5h pod/collect-profiles-27163485--1-46dbd 0/1 Terminating 0 3d12h pod/collect-profiles-27163605--1-swvbl 0/1 Terminating 0 3d10h pod/collect-profiles-27164040--1-qsbmh 0/1 Terminating 0 3d3h pod/collect-profiles-27164190--1-w4f2x 0/1 Terminating 0 3d pod/collect-profiles-27164220--1-hf56p 0/1 Terminating 0 3d pod/collect-profiles-27165030--1-m46zq 0/1 Terminating 0 2d10h pod/collect-profiles-27165075--1-g4l6j 0/1 Terminating 0 2d10h pod/collect-profiles-27166365--1-wdfvx 0/1 Terminating 0 36h pod/collect-profiles-27166455--1-dc7cg 0/1 Terminating 0 35h pod/collect-profiles-27166890--1-8ctrb 0/1 Terminating 0 3h56m pod/collect-profiles-27167340--1-9dq7x 0/1 Terminating 0 20h pod/collect-profiles-27167490--1-b6xd8 0/1 Terminating 0 17h pod/collect-profiles-27167715--1-7bwfq 0/1 Terminating 0 14h pod/collect-profiles-27168330--1-2cqqb 0/1 Error 0 168m pod/collect-profiles-27168330--1-6djvf 0/1 Error 0 164m pod/collect-profiles-27168330--1-9kng2 0/1 Error 0 3h1m pod/collect-profiles-27168330--1-c8xwq 0/1 Error 0 3h4m pod/collect-profiles-27168330--1-gczht 0/1 Error 0 171m pod/collect-profiles-27168330--1-rf4j6 0/1 Error 0 3h12m pod/collect-profiles-27168330--1-sfmhk 0/1 Error 0 175m pod/collect-profiles-27168525--1-6cfjp 0/1 Completed 0 44m pod/collect-profiles-27168540--1-kkzc6 0/1 Completed 0 29m pod/collect-profiles-27168555--1-pgrdq 0/1 Completed 0 14m pod/olm-operator-7f89d88f77-c6gzz 1/1 Running 38 5d pod/package-server-manager-77f8d8b554-v7pkn 1/1 Running 0 7h21m pod/packageserver-958484549-9dnkz 1/1 Running 38 5d pod/packageserver-958484549-nngpv 1/1 Running 41 5d NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/catalog-operator-metrics ClusterIP 172.30.85.212 <none> 8443/TCP 7d10h service/olm-operator-metrics ClusterIP 172.30.204.130 <none> 8443/TCP 7d10h service/packageserver-service ClusterIP 172.30.112.254 <none> 5443/TCP 37h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/catalog-operator 1/1 1 1 7d10h deployment.apps/olm-operator 1/1 1 1 7d10h deployment.apps/package-server-manager 1/1 1 1 7d10h deployment.apps/packageserver 2/2 2 2 7d10h NAME DESIRED CURRENT READY AGE replicaset.apps/catalog-operator-7d866fcb7b 1 1 1 7d10h replicaset.apps/olm-operator-7f89d88f77 1 1 1 7d10h replicaset.apps/package-server-manager-77f8d8b554 1 1 1 7d10h replicaset.apps/packageserver-958484549 2 2 2 7d10h NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE cronjob.batch/collect-profiles */15 * * * * False 0 14m 7d10h NAME COMPLETIONS DURATION AGE job.batch/collect-profiles-27168330 0/1 3h59m 3h59m job.batch/collect-profiles-27168525 1/1 15s 44m job.batch/collect-profiles-27168540 1/1 12s 29m job.batch/collect-profiles-27168555 1/1 10s 14m [root@rdr-cicd-e6b7-mon01-bastion-0 ~]#
I did some more digging. Found a tool to snoop on exec(). https://github.com/iovisor/bcc/blob/master/tools/execsnoop.py With a compiled version of the tool, I was able to correlate the new processes to the contents of /run/crio/exec-pid-dir: https://github.com/iovisor/bcc/blob/master/tools/execsnoop.py [root@rdr-cicd-e6b7-mon01-master-1 execsnoop]# ./execsnoop In file included from <built-in>:2: In file included from /virtual/include/bcc/bpf.h:12: In file included from include/linux/types.h:6: In file included from include/uapi/linux/types.h:14: In file included from include/uapi/linux/posix_types.h:5: In file included from include/linux/stddef.h:5: In file included from include/uapi/linux/stddef.h:2: In file included from include/linux/compiler_types.h:74: include/linux/compiler-clang.h:25:9: warning: '__no_sanitize_address' macro redefined [-Wmacro-redefined] #define __no_sanitize_address ^ include/linux/compiler-gcc.h:213:9: note: previous definition is here #define __no_sanitize_address __attribute__((no_sanitize_address)) ^ 1 warning generated. PCOMM PID PPID RET ARGS ldd 3733034 2553 0 /usr/bin/ldd /usr/bin/crio ld64.so.2 3733035 3733034 0 /lib64/ld64.so.2 --verify /usr/bin/crio ld64.so.2 3733038 3733037 0 /lib64/ld64.so.2 /usr/bin/crio sh 3733039 5068 0 /usr/bin/awk -F = '/partition_id/ { print $2 }' /proc/ppc64/lparcfg awk 3733039 5068 0 /usr/bin/awk -F = /partition_id/ { print $2 } /proc/ppc64/lparcfg sh 3733040 5068 0 /usr/bin/awk -F = '/partition_id/ { print $2 }' /proc/ppc64/lparcfg awk 3733040 5068 0 /usr/bin/awk -F = /partition_id/ { print $2 } /proc/ppc64/lparcfg sh 3733041 5068 0 /usr/bin/awk -F = '/partition_id/ { print $2 }' /proc/ppc64/lparcfg awk 3733041 5068 0 /usr/bin/awk -F = /partition_id/ { print $2 } /proc/ppc64/lparcfg sh 3733042 5313 0 /usr/bin/awk -F = '/partition_id/ { print $2 }' /proc/ppc64/lparcfg awk 3733042 5313 0 /usr/bin/awk -F = /partition_id/ { print $2 } /proc/ppc64/lparcfg sh 3733043 5313 0 /usr/bin/awk -F = '/partition_id/ { print $2 }' /proc/ppc64/lparcfg awk 3733043 5313 0 /usr/bin/awk -F = /partition_id/ { print $2 } /proc/ppc64/lparcfg sh 3733044 5313 0 /usr/bin/awk -F = '/partition_id/ { print $2 }' /proc/ppc64/lparcfg awk 3733044 5313 0 /usr/bin/awk -F = /partition_id/ { print $2 } /proc/ppc64/lparcfg md5sum 3733046 3733045 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/token awk 3733047 3733045 0 /usr/bin/awk {print $1} md5sum 3733049 3733048 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/ca.crt awk 3733050 3733048 0 /usr/bin/awk {print $1} sleep 3733051 3709 0 /usr/bin/sleep 1 md5sum 3733053 3733052 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/token awk 3733054 3733052 0 /usr/bin/awk {print $1} md5sum 3733056 3733055 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/ca.crt awk 3733057 3733055 0 /usr/bin/awk {print $1} sleep 3733058 3709 0 /usr/bin/sleep 1 runc 3733059 2553 0 /usr/bin/runc --root /run/runc exec --pid-file /var/run/crio/exec-pid-dir/1b2a759d965ccb9678cbae7b0dafde9537667d062555291284a1f5d6a2312fe36d95a00c-82c2-4fbe-b015-b2ab89cf3303 --process /tmp/exec-process-074617211 1b2a759d965ccb9678cbae7b0dafde9537667d062555291284a1f5d6a2312fe3 exe 3733068 3733059 0 /proc/self/exe init test 3733070 3733059 0 /usr/bin/test -f /etc/cni/net.d/80-openshift-network.conf md5sum 3733077 3733076 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/token awk 3733078 3733076 0 /usr/bin/awk {print $1} md5sum 3733080 3733079 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/ca.crt awk 3733081 3733079 0 /usr/bin/awk {print $1} sleep 3733082 3709 0 /usr/bin/sleep 1 runc 3733083 2553 0 /usr/bin/runc --root /run/runc exec --pid-file /var/run/crio/exec-pid-dir/afb03a1f9dbb802fc1d1440388430462cfb60231b060aa3ebcbc249ace50931519341ee0-e912-4070-a4f8-14d9196f1352 --process /tmp/exec-process-197278878 afb03a1f9dbb802fc1d1440388430462cfb60231b060aa3ebcbc249ace509315 exe 3733093 3733083 0 /proc/self/exe init bash 3733095 3733083 0 /bin/bash -c set -xe\n\n# Unix sockets are used for health checks to ensure that the pod is reporting readiness of the etcd process\n# in this c etcdctl 3733101 3733095 0 /usr/bin/etcdctl --command-timeout=2s --dial-timeout=2s --endpoints=unixs://193.168.200.231:0 endpoint health -w json grep 3733102 3733095 0 /usr/bin/grep "health":true awk 3733112 3733110 0 /usr/bin/awk {print $1} md5sum 3733111 3733110 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/token md5sum 3733114 3733113 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/ca.crt awk 3733115 3733113 0 /usr/bin/awk {print $1} sleep 3733116 3709 0 /usr/bin/sleep 1 runc 3733117 2553 0 /usr/bin/runc --root /run/runc exec --pid-file /var/run/crio/exec-pid-dir/612b6a23ee3b71174d42a96117c527fef4e4d3f5800a1257216af2fb249f4664df311a28-07fb-4c11-9bb9-9ccdf005adc2 --process /tmp/exec-process-392734080 612b6a23ee3b71174d42a96117c527fef4e4d3f5800a1257216af2fb249f4664 exe 3733126 3733117 0 /proc/self/exe init sh 3733131 3733117 0 /bin/sh -c declare -r health_endpoint="https://localhost:2379/health"\ndeclare -r cert="/var/run/secrets/etcd-client/tls.crt"\ndeclare -r key grep 3733138 3733131 0 /usr/bin/grep "health":"true" curl 3733137 3733131 0 /usr/bin/curl --max-time 2 --silent --cert /var/run/secrets/etcd-client/tls.crt --key /var/run/secrets/etcd-client/tls.key --cacert /var/run/configmaps/etcd-ca/ca-bundle.crt https://localhost:2379/health md5sum 3733141 3733140 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/token awk 3733142 3733140 0 /usr/bin/awk {print $1} md5sum 3733144 3733143 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/ca.crt awk 3733145 3733143 0 /usr/bin/awk {print $1} sleep 3733146 3709 0 /usr/bin/sleep 1 md5sum 3733148 3733147 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/token awk 3733149 3733147 0 /usr/bin/awk {print $1} md5sum 3733151 3733150 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/ca.crt awk 3733152 3733150 0 /usr/bin/awk {print $1} sleep 3733153 3709 0 /usr/bin/sleep 1 md5sum 3733155 3733154 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/token awk 3733156 3733154 0 /usr/bin/awk {print $1} md5sum 3733158 3733157 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/ca.crt awk 3733159 3733157 0 /usr/bin/awk {print $1} sleep 3733160 3709 0 /usr/bin/sleep 1 runc 3733161 2553 0 /usr/bin/runc --root /run/runc exec --pid-file /var/run/crio/exec-pid-dir/1b2a759d965ccb9678cbae7b0dafde9537667d062555291284a1f5d6a2312fe37b92324c-0714-46c1-b517-3ab9786ca397 --process /tmp/exec-process-050607721 1b2a759d965ccb9678cbae7b0dafde9537667d062555291284a1f5d6a2312fe3 exe 3733169 3733161 0 /proc/self/exe init test 3733173 3733161 0 /usr/bin/test -f /etc/cni/net.d/80-openshift-network.conf md5sum 3733180 3733179 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/token awk 3733181 3733179 0 /usr/bin/awk {print $1} md5sum 3733183 3733182 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/ca.crt awk 3733184 3733182 0 /usr/bin/awk {print $1} sleep 3733185 3709 0 /usr/bin/sleep 1 runc 3733186 2553 0 /usr/bin/runc --root /run/runc exec --pid-file /var/run/crio/exec-pid-dir/afb03a1f9dbb802fc1d1440388430462cfb60231b060aa3ebcbc249ace5093158e423623-9b1d-45d4-b3f3-63eea92c797b --process /tmp/exec-process-598901766 afb03a1f9dbb802fc1d1440388430462cfb60231b060aa3ebcbc249ace509315 exe 3733195 3733186 0 /proc/self/exe init bash 3733197 3733186 0 /bin/bash -c set -xe\n\n# Unix sockets are used for health checks to ensure that the pod is reporting readiness of the etcd process\n# in this c etcdctl 3733203 3733197 0 /usr/bin/etcdctl --command-timeout=2s --dial-timeout=2s --endpoints=unixs://193.168.200.231:0 endpoint health -w json grep 3733204 3733197 0 /usr/bin/grep "health":true awk 3733213 3733211 0 /usr/bin/awk {print $1} md5sum 3733212 3733211 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/token md5sum 3733215 3733214 0 /usr/bin/md5sum /var/run/secrets/kubernetes.io/serviceaccount/ca.crt awk 3733216 3733214 0 /usr/bin/awk {print $1} sleep 3733217 3709 0 /usr/bin/sleep 1 runc 3733218 2553 0 /usr/bin/runc --root /run/runc exec --pid-file /var/run/crio/exec-pid-dir/612b6a23ee3b71174d42a96117c527fef4e4d3f5800a1257216af2fb249f4664fd437fe6-f5b3-4f39-86b7-efb35ddcaa25 --process /tmp/exec-process-131250605 612b6a23ee3b71174d42a96117c527fef4e4d3f5800a1257216af2fb249f4664 exe 3733226 3733218 0 /proc/self/exe init sh 3733230 3733218 0 /bin/sh -c declare -r health_endpoint="https://localhost:2379/health"\ndeclare -r cert="/var/run/secrets/etcd-client/tls.crt"\ndeclare -r key grep 3733238 3733230 0 /usr/bin/grep "health":"true" curl 3733237 3733230 0 /usr/bin/curl --max-time 2 --silent --cert /var/run/secrets/etcd-client/tls.crt --key /var/run/secrets/etcd-client/tls.key --cacert /var/run/configmaps/etcd-ca/ca-bundle.crt https://localhost:2379/health ^CTraceback (most recent call last): File "execsnoop.py", line 305, in <module> File "bcc/__init__.py", line 1445, in perf_buffer_poll KeyboardInterrupt During handling of the above exception, another exception occurred: Traceback (most recent call last): File "execsnoop.py", line 307, in <module> NameError: name 'exit' is not defined [3732956] Failed to execute script 'execsnoop' due to unhandled exception! [root@rdr-cicd-e6b7-mon01-master-1 execsnoop]# for i in `ls -t /run/crio/exec-pid-dir|head `; do cat /run/crio/exec-pid-dir/$i; echo ' '; done 3733230 3733197 3733173 3733131 3733095 3733070 3733017 3732984 3732958 3732896
The 3 processes seem to be related to these pods: [root@rdr-cicd-e6b7-mon01-master-1 containers]# ls *1b2a759* sdn-jfkcp_openshift-sdn_sdn-1b2a759d965ccb9678cbae7b0dafde9537667d062555291284a1f5d6a2312fe3.log [root@rdr-cicd-e6b7-mon01-master-1 containers]# ls *afb03* etcd-rdr-cicd-e6b7-mon01-master-1_openshift-etcd_etcd-afb03a1f9dbb802fc1d1440388430462cfb60231b060aa3ebcbc249ace509315.log [root@rdr-cicd-e6b7-mon01-master-1 containers]# ls -al *612b6* lrwxrwxrwx. 1 root root 112 Aug 27 21:41 etcd-quorum-guard-59664ffdd8-dwrvk_openshift-etcd_guard-612b6a23ee3b71174d42a96117c527fef4e4d3f5800a1257216af2fb249f4664.log -> /var/log/pods/openshift-etcd_etcd-quorum-guard-59664ffdd8-dwrvk_04dd2823-7bbd-450f-9c70-8e5fd3472e80/guard/1.log
A new cluster was setup with the latest build ( using same config ). Checked the health of the cluster after 2 and half days. The same issue reported above is seen. # oc version Client Version: 4.9.0-0.nightly-ppc64le-2021-08-26-032145 Server Version: 4.9.0-0.nightly-ppc64le-2021-08-26-032145 Kubernetes Version: v1.22.0-rc.0+33d0ffa [root@rdr-cicd-4dc3-mon01-bastion-0 ~]# du -ha /run | sort -n -r | head -n 10 450M /run 448K /run/cloud-init 443M /run/log/journal 443M /run/log 427M /run/log/journal/e294339a289840009764dff6139c67e9 320K /run/NetworkManager 192K /run/systemd/generator 192K /run/NetworkManager/devices 128K /run/rhsm 128K /run/named Checked the file inside /run/log/journal/e294339a289840009764dff6139c67e9 journalctl --file system error seen in the above file is : Aug 30 01:00:23 rdr-cicd-4dc3-mon01-bastion-0.redhat.com multipathd[22218]: checked 2 paths in 0.001672 secs Aug 30 01:00:23 rdr-cicd-4dc3-mon01-bastion-0.redhat.com sshd[195192]: error: kex_exchange_identification: Connection closed by remote host Aug 30 01:00:24 rdr-cicd-4dc3-mon01-bastion-0.redhat.com multipathd[22218]: tick (1.001697 secs) Aug 30 01:00:24 rdr-cicd-4dc3-mon01-bastion-0.redhat.com multipathd[22218]: open '/sys/devices/vio/30000008/host2/rport-2:0-0/target2:0:0/2:0:0:1/state' Aug 30 01:00:24 rdr-cicd-4dc3-mon01-bastion-0.redhat.com multipathd[22218]: sdj: path state = running Aug 30 01:00:24 rdr-cicd-4dc3-mon01-bastion-0.redhat.com multipathd[22218]: 8:144 : tur checker starting up Aug 30 01:00:24 rdr-cicd-4dc3-mon01-bastion-0.redhat.com multipathd[22218]: 8:144 : tur checker finished, state up Aug 30 01:00:24 rdr-cicd-4dc3-mon01-bastion-0.redhat.com multipathd[22218]: sdj: tur state = up Aug 30 01:00:24 rdr-cicd-4dc3-mon01-bastion-0.redhat.com multipathd[22218]: mpathb: update_multipath_strings Aug 30 01:00:24 rdr-cicd-4dc3-mon01-bastion-0.redhat.com multipathd[22218]: libdevmapper: ioctl/libdm-iface.c(1898): dm table mpathb [ noopencount flush ] [16384] (*1)
Created attachment 1818991 [details] Content of .journal file Attached file has the output of : journalctl --file system mentioned in comment 16.
This is being reported with 4.8.5 as well. i.e. Potential to be hit by customers who upgrade to the most recent release.
@prashanth found that this issue was introduced by https://github.com/cri-o/cri-o/pull/5136 And it is fixed/reverted by: https://github.com/cri-o/cri-o/pull/5245 https://github.com/cri-o/cri-o/pull/5262
We're asking the following questions to evaluate whether or not this bug warrants blocking an upgrade edge from either the previous X.Y or X.Y.Z. The ultimate goal is to avoid delivering an update which introduces new risk or reduces cluster functionality in any way. Sample answers are provided to give more context and the ImpactStatementRequested label has been added to this bug. When responding, please remove ImpactStatementRequested and set the ImpactStatementProposed label. The expectation is that the assignee answers these questions. Who is impacted? If we have to block upgrade edges based on this issue, which edges would need blocking? * example: Customers upgrading from 4.y.Z to 4.y+1.z running on GCP with thousands of namespaces, approximately 5% of the subscribed fleet * example: All customers upgrading from 4.y.z to 4.y+1.z fail approximately 10% of the time What is the impact? Is it serious enough to warrant blocking edges? * example: Up to 2 minute disruption in edge routing * example: Up to 90 seconds of API downtime * example: etcd loses quorum and you have to restore from backup How involved is remediation (even moderately serious impacts might be acceptable if they are easy to mitigate)? * example: Issue resolves itself after five minutes * example: Admin uses oc to fix things * example: Admin must SSH to hosts, restore from backups, or other non standard admin activities Is this a regression (if all previous versions were also vulnerable, updating to the new, vulnerable version does not increase exposure)? * example: No, it has always been like this we just never noticed * example: Yes, from 4.y.z to 4.y+1.z Or 4.y.z to 4.y.z+1
Straight from NEW -> VERIFIED like we had here after comment 19 skips some steps. Moving back to MODIFIED, so ART's Elliot sweeper can associate with an errata.
Also setting a priority, while I'm tidying bug state.
Also adding Needinfo for Manoj and Prashanth as they are closer to this bug regarding Comment 20
Reposting Trevor's summary. Who is impacted? If we have to block upgrade edges based on this issue, which edges would need blocking? * Updates from earlier releases into the 4.7.(>=24) or 4.8.(5 <= z <= 9) What is the impact? Is it serious enough to warrant blocking edges? * /run could fill up because of directories created for exec probes. How soon it could fill up depends on how many and how frequently the probes are running. /run filling up leads to Node running out of memory and going Not Ready. How involved is remediation (even moderately serious impacts might be acceptable if they are easy to mitigate)? * Reboot affected nodes as needed to keep the cluster healthy until you can update to a fixed release. Is this a regression (if all previous versions were also vulnerable, updating to the new, vulnerable version does not increase exposure)? * Yes: * 4.6.43 is unaffected, next 4.6.z to ship will be unless we fix things. * 4.7.24 - 4.7.28 include the regression. * 4.8.5 - 4.8.9 include the regression.
Based on the impact statement in comment 24, we've landed [1], blocking some edges into 4.7.24, 4.7.28, 4.8.5, and 4.8.9 to protect supported clusters from the CRI-O /run leak. The edges blocked protect supported releases from transitioning into the impacted releases (e.g. no 4.7.23 -> 4.8.9). We continue to recommend updates between impacted releases (e.g. 4.7.24 -> 4.8.9), because moving from one impacted release to another impacted release does not make things worse (in fact, the node reboots may help make things temporarily better). We do not manage edges that only exist in candidate channels, so those mostly stayed in place as well. [1]: https://github.com/openshift/cincinnati-graph-data/pull/1022
api.libvirt-ppc64le-0-0 - 4.6 [core@libvirt-ppc64le-0-0-7-4frf7-master-0 ~]$ cat /etc/redhat-release Red Hat Enterprise Linux CoreOS release 4.6 [core@libvirt-ppc64le-0-0-7-4frf7-master-0 ~]$ sudo find / -name exec-pid-dir 2>/dev/null [core@libvirt-ppc64le-0-0-7-4frf7-master-0 ~]$ api.libvirt-ppc64le-2-1 - 4.7 [core@libvirt-ppc64le-2-1-b-67wm2-master-0 ~]$ cat /etc/redhat-release Red Hat Enterprise Linux CoreOS release 4.7 [core@libvirt-ppc64le-2-1-b-67wm2-master-0 ~]$ sudo find / -name exec-pid-dir 2>/dev/null [core@libvirt-ppc64le-2-1-b-67wm2-master-0 ~]$ api.libvirt-ppc64le-0-3 - 4.8 [core@libvirt-ppc64le-0-3-0-ql4z8-master-0 ~]$ cat /etc/redhat-release Red Hat Enterprise Linux CoreOS release 4.8 [core@libvirt-ppc64le-0-3-0-ql4z8-master-0 ~]$ sudo find / -name exec-pid-dir 2>/dev/null /run/crio/exec-pid-dir [core@libvirt-ppc64le-0-3-0-ql4z8-master-0 ~]$ sudo ls -la /run/crio/exec-pid-dir total 0 drwxr-x---. 2 root root 40 Sep 2 18:19 . drwxr-xr-x. 4 root root 4780 Sep 2 18:49 .. [core@libvirt-ppc64le-0-3-0-ql4z8-master-1 ~]$ sudo ls -la /run/crio/exec-pid-dir total 0 drwxr-x---. 2 root root 40 Sep 2 18:19 . drwxr-xr-x. 4 root root 5060 Sep 2 18:51 .. [core@libvirt-ppc64le-0-3-0-ql4z8-master-2 ~]$ sudo ls -la /run/crio/exec-pid-dir total 0 drwxr-x---. 2 root root 40 Sep 2 18:19 . drwxr-xr-x. 4 root root 4460 Sep 2 18:51 .. api.libvirt-ppc64le-1-0 - 4.9 [core@libvirt-ppc64le-1-0-6-4fh7c-master-0 ~]$ cat /etc/redhat-release Red Hat Enterprise Linux CoreOS release 4.9 [core@libvirt-ppc64le-1-0-6-4fh7c-master-0 ~]$ sudo find / -name exec-pid-dir 2>/dev/null /run/crio/exec-pid-dir [core@libvirt-ppc64le-1-0-6-4fh7c-master-0 ~]$ sudo ls -la /run/crio/exec-pid-dir total 0 drwxr-x---. 2 root root 40 Sep 2 18:19 . drwxr-xr-x. 4 root root 3820 Sep 2 18:41 .. [core@libvirt-ppc64le-1-0-6-4fh7c-master-1 ~]$ sudo ls -la /run/crio/exec-pid-dir total 0 drwxr-x---. 2 root root 40 Sep 2 18:18 . drwxr-xr-x. 4 root root 3080 Sep 2 18:41 .. [core@libvirt-ppc64le-1-0-6-4fh7c-master-2 ~]$ sudo ls -la /run/crio/exec-pid-dir total 0 drwxr-x---. 2 root root 40 Sep 2 18:18 . drwxr-xr-x. 4 root root 2760 Sep 2 18:42 ..
A workaround for anyone who cannot upgrade is documented in https://access.redhat.com/solutions/6304881
We updated from 4.7.2 to 4.7.26 using the candidate channel and then (right away) from 4.7.26 to 4.8.7 using the candidate channel. Never had any issues upgrading except one worker node needed to be restarted manually to proceed with the machine set upgrade. Next, we went from 4.8.7 to 4.8.9. Currently, we are on 4.8.10. What is the status of this bug in 4.8.10?
Also, even though we had 4.7.26 running for two weeks and 4.8.9 for two days, we do not see the problem on any of our (AWS) Nodes e.g. sh-4.4# chroot /host sh-4.4# ls -l /run/crio/exec-pid-dir total 0 sh-4.4# exit Seems good. So for me, it is really hard to ascertain how universal this problem is.
(In reply to sven.thoms from comment #31) > What is the status of this bug in 4.8.10? 4.8-specific discussion should go in the 4.8.z bug (which shipped in 4.8.10 [1]). [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1999645#c13
For whatever it is worth, we are seeing the problem on OCP 4.6.43 running on AWS. ls -la /run/crio/exec-pid-dir | wc -l 7824683 df -ki /run/crio Filesystem Inodes IUsed IFree IUse% Mounted on tmpfs 8142826 7856422 286404 97% /run crio --version crio version 1.19.3-9.rhaos4.6.gitc8a7d88.el8 Version: 1.19.3-9.rhaos4.6.gitc8a7d88.el8 GoVersion: go1.15.14 Compiler: gc Platform: linux/amd64 Linkmode: dynamic
uh oh, it seems we were incorrect about impact. 4.6.43 does indeed have this problem. However, I have double checked that 4.6.44 has the fix
We've extended the graph-data blocks from comment 25 to include * -> 4.6.43 [1] based on comment 35 and comment 36. Thanks for the report :) [1]: https://github.com/openshift/cincinnati-graph-data/pull/1085
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759