Created attachment 1794553 [details] must-gather from cluster where this occurred Description of problem: Pod "cluster-version-operator-89bf5cdb5-4qhhh" in openshift-cluster-version namespace was not handled by the workload partitioning pod mutation logic. A warning was added to the pod: apiVersion: v1 kind: Pod metadata: annotations: workload.openshift.io/warning: only single-node clusters support workload partitioning Version-Release number of selected component (if applicable): 4.8.0-0.nightly-2021-06-24-222938 How reproducible: unknown Steps to Reproduce: 1. Cluster installed 2. "oc describe node" shows 20m CPU requests for this pod 3. Actual results: openshift-cluster-version cluster-version-operator-89bf5cdb5-4qhhh 20m (0%) 0 (0%) 50Mi (0%) 0 (0%) 4h Expected results: openshift-cluster-version cluster-version-operator-89bf5cdb5-4qhhh 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 4h Additional info:
Can you please provide the installer debug log?
First tried to study https://github.com/openshift/enhancements/blob/master/enhancements/workload-partitioning/management-workload-partitioning.md to get the background (due to it is Node scope, I'm moving to Node component Autoscaler subcomponent but I'll continue to verify it). Second, tried in old unfixed version like 4.8.0-0.nightly-2021-07-02-143616 in non-SNO env: oc get po -n openshift-cluster-version -o yaml # can easily see the incorrect warning: metadata: annotations: workload.openshift.io/warning: only single-node clusters support workload partitioning name: cluster-version-operator-78f9799669-knfsz oc describe no POD_NODE | grep cluster-version-operator # can easily see 20m CPU request for the CVO pod Then tried latest 4.9 non-SNO env (4.9.0-0.nightly-2021-07-04-140102 as of now), the issue still exists. Then checked 4.9.0-0.nightly-2021-07-04-140102, its last o/k commit is 0228142e087: oc adm release info --commits registry.ci.openshift.org/ocp/release:4.9.0-0.nightly-2021-07-04-140102 | grep hyperkube hyperkube https://github.com/openshift/kubernetes 0228142e087... https://github.com/openshift/kubernetes/blob/0228142e087/openshift-kube-apiserver/admission/autoscaling/managementcpusoverride/admission.go#L183-L186 already contains the fix of PR. Thus moving back to ASSIGNED
I think the title of the bug misleads you, the bug should be verified on the SNO with the workload partitioning enabled. Can you please verify on the SNO environment with the workload partition enabled?
(In reply to Artyom from comment #4) Thanks for clarification. Then it is better to ask help from the QE colleague from team of the workload partitioning feature. Let me update.
Checked on SNO setup with workload partitioning configured during cluster deployment. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-rc.0 True False 3h8m Cluster version is 4.9.0-rc.0 $ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master-0.cluster1.savanna.lab.eng.rdu2.redhat.com Ready master,worker 3h26m v1.22.0-rc.0+75ee307 10.1.190.12 <none> Red Hat Enterprise Linux CoreOS 49.84.202109041651-0 (Ootpa) 4.18.0-305.17.1.rt7.89.el8_4.x86_64 cri-o://1.22.0-68.rhaos4.9.git011c10a.el8 $ oc describe mc 02-master-workload-partitioning Name: 02-master-workload-partitioning Namespace: Labels: machineconfiguration.openshift.io/role=master Annotations: <none> API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig Metadata: Creation Timestamp: 2021-09-14T12:55:06Z Generation: 1 Managed Fields: API Version: machineconfiguration.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:labels: .: f:machineconfiguration.openshift.io/role: f:spec: .: f:config: .: f:ignition: .: f:version: f:storage: .: f:files: Manager: cluster-bootstrap Operation: Update Time: 2021-09-14T12:55:06Z Resource Version: 1874 UID: 330a580d-b29a-4fa6-b872-7add31619671 Spec: Config: Ignition: Version: 3.2.0 Storage: Files: Contents: Source: data:text/plain;charset=utf-8;base64,W2NyaW8ucnVudGltZS53b3JrbG9hZHMubWFuYWdlbWVudF0KYWN0aXZhdGlvbl9hbm5vdGF0aW9uID0gInRhcmdldC53b3JrbG9hZC5vcGVuc2hpZnQuaW8vbWFuYWdlbWVudCIKYW5ub3RhdGlvbl9wcmVmaXggPSAicmVzb3VyY2VzLndvcmtsb2FkLm9wZW5zaGlmdC5pbyIKcmVzb3VyY2VzID0geyAiY3B1c2hhcmVzIiA9IDAsICJjcHVzZXQiID0gIjAtMSw0MC00MSIgfQo= Mode: 420 Overwrite: true Path: /etc/crio/crio.conf.d/01-workload-partitioning User: Name: root Contents: Source: data:text/plain;charset=utf-8;base64,ewogICJtYW5hZ2VtZW50IjogewogICAgImNwdXNldCI6ICIwLTEsNDAtNDEiCiAgfQp9Cg== Mode: 420 Overwrite: true Path: /etc/kubernetes/openshift-workload-pinning User: Name: root Events: <none> $ oc describe node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com Name: master-0.cluster1.savanna.lab.eng.rdu2.redhat.com Roles: master,worker Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=master-0.cluster1.savanna.lab.eng.rdu2.redhat.com kubernetes.io/os=linux node-role.kubernetes.io/master= node-role.kubernetes.io/worker= node.openshift.io/os_id=rhcos Annotations: k8s.ovn.org/host-addresses: ["10.1.190.12","10.1.190.40","10.1.190.44","172.22.0.166"] k8s.ovn.org/l3-gateway-config: {"default":{"mode":"shared","interface-id":"br-ex_master-0.cluster1.savanna.lab.eng.rdu2.redhat.com","mac-address":"98:03:9b:8e:86:c4","ip... k8s.ovn.org/node-chassis-id: a7ddf241-881e-45f1-8809-8541528be673 k8s.ovn.org/node-mgmt-port-mac-address: 56:ce:0e:59:6f:09 k8s.ovn.org/node-primary-ifaddr: {"ipv4":"10.1.190.12/23"} k8s.ovn.org/node-subnets: {"default":"10.128.0.0/23"} k8s.ovn.org/topology-version: 4 machineconfiguration.openshift.io/controlPlaneTopology: SingleReplica machineconfiguration.openshift.io/currentConfig: rendered-master-c61924ab4910008fe9f7d47ea1d19dad machineconfiguration.openshift.io/desiredConfig: rendered-master-c61924ab4910008fe9f7d47ea1d19dad machineconfiguration.openshift.io/reason: machineconfiguration.openshift.io/ssh: accessed machineconfiguration.openshift.io/state: Done sriovnetwork.openshift.io/state: Idle volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Tue, 14 Sep 2021 18:40:24 +0530 Taints: <none> Unschedulable: false Lease: HolderIdentity: master-0.cluster1.savanna.lab.eng.rdu2.redhat.com AcquireTime: <unset> RenewTime: Tue, 14 Sep 2021 22:07:34 +0530 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Tue, 14 Sep 2021 22:02:36 +0530 Tue, 14 Sep 2021 18:40:24 +0530 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Tue, 14 Sep 2021 22:02:36 +0530 Tue, 14 Sep 2021 18:40:24 +0530 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Tue, 14 Sep 2021 22:02:36 +0530 Tue, 14 Sep 2021 18:40:24 +0530 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Tue, 14 Sep 2021 22:02:36 +0530 Tue, 14 Sep 2021 18:46:10 +0530 KubeletReady kubelet is posting ready status Addresses: InternalIP: 10.1.190.12 Hostname: master-0.cluster1.savanna.lab.eng.rdu2.redhat.com Capacity: cpu: 80 ephemeral-storage: 1561964428Ki hugepages-1Gi: 32Gi hugepages-2Mi: 0 management.workload.openshift.io/cores: 80k memory: 394645704Ki openshift.io/du_mh: 8 pods: 250 Allocatable: cpu: 76 ephemeral-storage: 1561964428Ki hugepages-1Gi: 32Gi hugepages-2Mi: 0 management.workload.openshift.io/cores: 80k memory: 359964872Ki openshift.io/du_mh: 8 pods: 250 System Info: Machine ID: 8869ab0904fb429f9e338d90028ff69c System UUID: 4c4c4544-0048-3510-8051-b3c04f505832 Boot ID: e682eec2-a8f3-410a-92dc-8a424477f3c8 Kernel Version: 4.18.0-305.17.1.rt7.89.el8_4.x86_64 OS Image: Red Hat Enterprise Linux CoreOS 49.84.202109041651-0 (Ootpa) Operating System: linux Architecture: amd64 Container Runtime Version: cri-o://1.22.0-68.rhaos4.9.git011c10a.el8 Kubelet Version: v1.22.0-rc.0+75ee307 Kube-Proxy Version: v1.22.0-rc.0+75ee307 Non-terminated Pods: (96 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- open-cluster-management-agent-addon klusterlet-addon-appmgr-68c6c47fc6-dh6p7 0 (0%) 0 (0%) 128Mi (0%) 2Gi (0%) 3h1m open-cluster-management-agent-addon klusterlet-addon-operator-6f557446b6-v8xmw 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h4m open-cluster-management-agent-addon klusterlet-addon-policyctrl-config-policy-779cd698bb-db9wt 0 (0%) 0 (0%) 128Mi (0%) 256Mi (0%) 3h1m open-cluster-management-agent-addon klusterlet-addon-policyctrl-framework-6879ffdf9b-jmscq 0 (0%) 0 (0%) 384Mi (0%) 768Mi (0%) 3h1m open-cluster-management-agent-addon klusterlet-addon-workmgr-5d8d97bcd5-2l8kq 0 (0%) 0 (0%) 128Mi (0%) 512Mi (0%) 3h1m open-cluster-management-agent klusterlet-7dbdb54759-mfvcg 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h6m open-cluster-management-agent klusterlet-registration-agent-6bd475b87c-gqdrt 0 (0%) 0 (0%) 128Mi (0%) 0 (0%) 3h5m open-cluster-management-agent klusterlet-work-agent-6bfdcc9ccf-vbsm8 0 (0%) 0 (0%) 128Mi (0%) 0 (0%) 3h5m openshift-apiserver-operator openshift-apiserver-operator-67475f6559-7p6x8 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m openshift-apiserver apiserver-7584f94579-99j9d 0 (0%) 0 (0%) 250Mi (0%) 0 (0%) 3h14m openshift-authentication-operator authentication-operator-697f847cb-6n7mz 0 (0%) 0 (0%) 200Mi (0%) 0 (0%) 3h26m openshift-authentication oauth-openshift-855d946b6b-dpptv 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h11m openshift-cloud-controller-manager-operator cluster-cloud-controller-manager-operator-6c45546cfb-kzhmk 0 (0%) 0 (0%) 75Mi (0%) 0 (0%) 3h15m openshift-cloud-credential-operator cloud-credential-operator-765479c5d8-8cmsh 0 (0%) 0 (0%) 170Mi (0%) 0 (0%) 3h26m openshift-cluster-machine-approver machine-approver-5bc975c7f7-lx997 0 (0%) 0 (0%) 70Mi (0%) 0 (0%) 3h26m openshift-cluster-node-tuning-operator cluster-node-tuning-operator-6544b7f674-c8x6f 0 (0%) 0 (0%) 20Mi (0%) 0 (0%) 3h26m openshift-cluster-node-tuning-operator tuned-lk7m2 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h21m openshift-cluster-samples-operator cluster-samples-operator-754cdd9744-m5982 0 (0%) 0 (0%) 100Mi (0%) 0 (0%) 3h15m openshift-cluster-storage-operator cluster-storage-operator-6554f956d8-tdgsh 0 (0%) 0 (0%) 20Mi (0%) 0 (0%) 3h15m openshift-cluster-storage-operator csi-snapshot-controller-97dfb65b7-x5ggx 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h21m openshift-cluster-storage-operator csi-snapshot-controller-operator-7b87d997b7-gk7f2 0 (0%) 0 (0%) 65Mi (0%) 0 (0%) 3h26m openshift-cluster-storage-operator csi-snapshot-webhook-987b4bc98-t5fd7 0 (0%) 0 (0%) 20Mi (0%) 0 (0%) 3h21m openshift-cluster-version cluster-version-operator-7cd8b7f756-5vrs5 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m openshift-config-operator openshift-config-operator-6d5b45f545-sh98v 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m openshift-console-operator console-operator-6b6f56c57f-bwrhx 0 (0%) 0 (0%) 100Mi (0%) 0 (0%) 3h15m openshift-controller-manager-operator openshift-controller-manager-operator-5cdb6958f5-6m4vh 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m openshift-controller-manager controller-manager-xm5p2 0 (0%) 0 (0%) 100Mi (0%) 0 (0%) 3h15m openshift-dns-operator dns-operator-67968cf9f9-vhgtr 0 (0%) 0 (0%) 69Mi (0%) 0 (0%) 3h26m openshift-dns dns-default-c9vkf 0 (0%) 0 (0%) 110Mi (0%) 0 (0%) 3h22m openshift-dns node-resolver-5f4qv 0 (0%) 0 (0%) 21Mi (0%) 0 (0%) 3h22m openshift-etcd-operator etcd-operator-88785ddc-g55cb 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m openshift-etcd etcd-master-0.cluster1.savanna.lab.eng.rdu2.redhat.com 0 (0%) 0 (0%) 930Mi (0%) 0 (0%) 3h17m openshift-image-registry cluster-image-registry-operator-67586bb65b-7jzlf 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h15m openshift-image-registry node-ca-56h5b 0 (0%) 0 (0%) 10Mi (0%) 0 (0%) 3h12m openshift-ingress-canary ingress-canary-qrjtk 0 (0%) 0 (0%) 20Mi (0%) 0 (0%) 3h12m openshift-ingress-operator ingress-operator-9fcf4fcbf-g8fwr 0 (0%) 0 (0%) 96Mi (0%) 0 (0%) 3h15m openshift-ingress router-default-5757cfd87-d8g9g 0 (0%) 0 (0%) 256Mi (0%) 0 (0%) 3h12m openshift-insights insights-operator-5f8b67548b-wnzpl 0 (0%) 0 (0%) 30Mi (0%) 0 (0%) 3h15m openshift-kube-apiserver-operator kube-apiserver-operator-565b5d9bbc-t7gjw 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m openshift-kube-apiserver kube-apiserver-master-0.cluster1.savanna.lab.eng.rdu2.redhat.com 0 (0%) 0 (0%) 1224Mi (0%) 0 (0%) 3h10m openshift-kube-controller-manager-operator kube-controller-manager-operator-86bb8c68fb-mljnl 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m openshift-kube-controller-manager kube-controller-manager-master-0.cluster1.savanna.lab.eng.rdu2.redhat.com 0 (0%) 0 (0%) 500Mi (0%) 0 (0%) 3h11m openshift-kube-scheduler-operator openshift-kube-scheduler-operator-f5c467697-cvkb9 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m openshift-kube-scheduler openshift-kube-scheduler-master-0.cluster1.savanna.lab.eng.rdu2.redhat.com 0 (0%) 0 (0%) 150Mi (0%) 0 (0%) 3h11m openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator-5d69c9c8bf-h6gzd 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m openshift-kube-storage-version-migrator migrator-798cb9b757-bvlbn 0 (0%) 0 (0%) 200Mi (0%) 0 (0%) 3h22m openshift-local-storage local-storage-operator-7f6477ddfd-fhxm4 0 (0%) 0 (0%) 0 (0%) 0 (0%) 166m openshift-logging cluster-logging-operator-5df48cc674-qprsj 0 (0%) 0 (0%) 0 (0%) 0 (0%) 166m openshift-logging fluentd-9sl7x 0 (0%) 0 (0%) 736Mi (0%) 736Mi (0%) 150m openshift-machine-api cluster-autoscaler-operator-766bb4796f-wpdzx 0 (0%) 0 (0%) 70Mi (0%) 0 (0%) 3h15m openshift-machine-api cluster-baremetal-operator-79f5f7c994-6jpdc 0 (0%) 0 (0%) 70Mi (0%) 0 (0%) 3h15m openshift-machine-api machine-api-operator-bd8ffc766-659fw 0 (0%) 0 (0%) 70Mi (0%) 0 (0%) 3h15m openshift-machine-config-operator machine-config-controller-655f487b68-jlcpn 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h21m openshift-machine-config-operator machine-config-daemon-gqrhc 0 (0%) 0 (0%) 100Mi (0%) 0 (0%) 3h22m openshift-machine-config-operator machine-config-operator-78c44d9f9c-znwcv 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m openshift-machine-config-operator machine-config-server-7pz4p 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h21m openshift-marketplace cluster-logging-operator-brqfl 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 175m openshift-marketplace local-storage-operator-48sw9 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h openshift-marketplace marketplace-operator-6d49688f8c-x8cfr 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m openshift-marketplace performance-addon-operator-9trqk 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 175m openshift-marketplace ptp-operator-xvqr9 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 174m openshift-marketplace sriov-network-operator-r5bh6 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h openshift-monitoring alertmanager-main-0 0 (0%) 0 (0%) 105Mi (0%) 0 (0%) 3h12m openshift-monitoring cluster-monitoring-operator-85f6566b9-84hvz 0 (0%) 0 (0%) 95Mi (0%) 0 (0%) 3h26m openshift-monitoring grafana-5bf796858d-7dd6d 0 (0%) 0 (0%) 84Mi (0%) 0 (0%) 3h12m openshift-monitoring kube-state-metrics-7c7fb896db-5k879 0 (0%) 0 (0%) 110Mi (0%) 0 (0%) 3h21m openshift-monitoring node-exporter-d5c2c 0 (0%) 0 (0%) 47Mi (0%) 0 (0%) 3h21m openshift-monitoring openshift-state-metrics-69b88f859f-mhlg8 0 (0%) 0 (0%) 72Mi (0%) 0 (0%) 3h21m openshift-monitoring prometheus-adapter-5fb75fd796-bgspj 0 (0%) 0 (0%) 40Mi (0%) 0 (0%) 3h15m openshift-monitoring prometheus-k8s-0 0 (0%) 0 (0%) 1119Mi (0%) 0 (0%) 3h12m openshift-monitoring prometheus-operator-7c8d859ddc-tsrbd 0 (0%) 0 (0%) 165Mi (0%) 0 (0%) 3h22m openshift-monitoring thanos-querier-5779d46758-c5vhg 0 (0%) 0 (0%) 77Mi (0%) 0 (0%) 3h12m openshift-multus multus-4nd96 0 (0%) 0 (0%) 65Mi (0%) 0 (0%) 3h24m openshift-multus multus-additional-cni-plugins-tfms6 0 (0%) 0 (0%) 10Mi (0%) 0 (0%) 3h24m openshift-multus multus-admission-controller-99f8p 0 (0%) 0 (0%) 70Mi (0%) 0 (0%) 3h22m openshift-multus network-metrics-daemon-pqhp8 0 (0%) 0 (0%) 120Mi (0%) 0 (0%) 3h24m openshift-network-diagnostics network-check-source-6b4b6f7445-k5zb8 0 (0%) 0 (0%) 40Mi (0%) 0 (0%) 3h24m openshift-network-diagnostics network-check-target-qqnkg 0 (0%) 0 (0%) 15Mi (0%) 0 (0%) 3h24m openshift-network-operator network-operator-5b9cd89648-qk9bx 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h26m openshift-oauth-apiserver apiserver-56d7684bb6-l7gs2 0 (0%) 0 (0%) 200Mi (0%) 0 (0%) 3h22m openshift-operator-lifecycle-manager catalog-operator-764dd45b5d-674d9 0 (0%) 0 (0%) 80Mi (0%) 0 (0%) 3h15m openshift-operator-lifecycle-manager olm-operator-794d77cdb6-krnjd 0 (0%) 0 (0%) 160Mi (0%) 0 (0%) 3h15m openshift-operator-lifecycle-manager package-server-manager-fcd7dc6d-wrw49 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h15m openshift-operator-lifecycle-manager packageserver-b8f685656-68fnl 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h13m openshift-ovn-kubernetes ovnkube-master-swtz9 0 (0%) 0 (0%) 1520Mi (0%) 0 (0%) 3h24m openshift-ovn-kubernetes ovnkube-node-clj9p 0 (0%) 0 (0%) 640Mi (0%) 0 (0%) 3h24m openshift-performance-addon-operator performance-operator-556c5b77db-8gtng 0 (0%) 0 (0%) 25Mi (0%) 0 (0%) 165m openshift-ptp linuxptp-daemon-sxtkg 0 (0%) 0 (0%) 70Mi (0%) 0 (0%) 125m openshift-ptp ptp-operator-54c64b6c5f-zwvc7 0 (0%) 0 (0%) 100Mi (0%) 0 (0%) 126m openshift-service-ca-operator service-ca-operator-5b7bd8c945-tv8rx 0 (0%) 0 (0%) 80Mi (0%) 0 (0%) 3h26m openshift-service-ca service-ca-64cf77d8f8-48r4s 0 (0%) 0 (0%) 120Mi (0%) 0 (0%) 3h22m openshift-sriov-network-operator network-resources-injector-hsc77 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 125m openshift-sriov-network-operator operator-webhook-mdnlc 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 125m openshift-sriov-network-operator sriov-device-plugin-x6gmg 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 112m openshift-sriov-network-operator sriov-network-config-daemon-25v6k 0 (0%) 0 (0%) 120Mi (0%) 0 (0%) 125m openshift-sriov-network-operator sriov-network-operator-869cbff7bc-hz92p 0 (0%) 0 (0%) 0 (0%) 0 (0%) 126m Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 0 (0%) 0 (0%) memory 13625Mi (3%) 4320Mi (1%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) management.workload.openshift.io/cores 2871 2871 openshift.io/du_mh 0 0 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal NodeReady 3h22m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeReady Normal NodeAllocatableEnforced 3h22m kubelet Updated Node Allocatable limit across pods Normal Starting 3h22m kubelet Starting kubelet. Normal NodeHasSufficientMemory 3h22m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 3h22m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasNoDiskPressure Normal NodeNotReady 3h22m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeNotReady Normal NodeHasSufficientPID 3h22m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientPID Normal NodeReady 3h22m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeReady Normal NodeNotReady 3h21m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeNotReady Normal Starting 3h21m kubelet Starting kubelet. Normal NodeHasSufficientMemory 3h21m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientMemory Normal NodeHasSufficientPID 3h21m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientPID Normal NodeHasNoDiskPressure 3h21m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasNoDiskPressure Normal NodeAllocatableEnforced 3h21m kubelet Updated Node Allocatable limit across pods Normal NodeReady 3h21m kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeReady Normal Starting 177m kubelet Starting kubelet. Normal NodeAllocatableEnforced 177m kubelet Updated Node Allocatable limit across pods Normal NodeHasSufficientMemory 177m (x8 over 177m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 177m (x8 over 177m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 177m (x7 over 177m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientPID Normal NodeAllocatableEnforced 170m kubelet Updated Node Allocatable limit across pods Normal NodeHasSufficientMemory 170m (x8 over 170m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 170m (x8 over 170m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 170m (x7 over 170m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientPID Normal Starting 170m kubelet Starting kubelet. Normal Starting 161m kubelet Starting kubelet. Normal NodeHasNoDiskPressure 161m (x8 over 161m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 161m (x7 over 161m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientPID Normal NodeAllocatableEnforced 161m kubelet Updated Node Allocatable limit across pods Normal NodeHasSufficientMemory 161m (x8 over 161m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientMemory Normal NodeAllocatableEnforced 153m kubelet Updated Node Allocatable limit across pods Normal Starting 153m kubelet Starting kubelet. Normal NodeHasSufficientMemory 153m (x8 over 153m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientMemory Normal NodeHasSufficientPID 153m (x7 over 153m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientPID Normal NodeHasNoDiskPressure 153m (x8 over 153m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasNoDiskPressure Normal Starting 116m kubelet Starting kubelet. Normal NodeHasSufficientMemory 116m (x8 over 116m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 116m (x8 over 116m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 116m (x7 over 116m) kubelet Node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com status is now: NodeHasSufficientPID Normal NodeAllocatableEnforced 116m kubelet Updated Node Allocatable limit across pods $ oc describe node master-0.cluster1.savanna.lab.eng.rdu2.redhat.com | grep -i openshift-cluster-version openshift-cluster-version cluster-version-operator-7cd8b7f756-5vrs5 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 3h27m $ oc describe pod cluster-version-operator-7cd8b7f756-5vrs5 -n openshift-cluster-version Name: cluster-version-operator-7cd8b7f756-5vrs5 Namespace: openshift-cluster-version Priority: 2000000000 Priority Class Name: system-cluster-critical Node: master-0.cluster1.savanna.lab.eng.rdu2.redhat.com/10.1.190.12 Start Time: Tue, 14 Sep 2021 18:41:03 +0530 Labels: k8s-app=cluster-version-operator pod-template-hash=7cd8b7f756 Annotations: resources.workload.openshift.io/cluster-version-operator: {"cpushares": 20} target.workload.openshift.io/management: {"effect":"PreferredDuringScheduling"} Status: Running IP: 10.1.190.12 IPs: IP: 10.1.190.12 Controlled By: ReplicaSet/cluster-version-operator-7cd8b7f756 Containers: cluster-version-operator: Container ID: cri-o://0f7982f0f4c90a16f5df9d16ebf50ed3e3ebd58b9c08eb1a352d4b02d4a3e19e Image: registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca Image ID: registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca Port: <none> Host Port: <none> Args: start --release-image=registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca --enable-auto-update=false --enable-default-cluster-version=true --listen=0.0.0.0:9099 --serving-cert-file=/etc/tls/serving-cert/tls.crt --serving-key-file=/etc/tls/serving-cert/tls.key --v=5 State: Running Started: Tue, 14 Sep 2021 20:11:39 +0530 Ready: True Restart Count: 5 Limits: management.workload.openshift.io/cores: 20 Requests: management.workload.openshift.io/cores: 20 memory: 50Mi Environment: KUBERNETES_SERVICE_PORT: 6443 KUBERNETES_SERVICE_HOST: 127.0.0.1 NODE_NAME: (v1:spec.nodeName) CLUSTER_PROFILE: self-managed-high-availability Mounts: /etc/cvo/updatepayloads from etc-cvo-updatepayloads (ro) /etc/ssl/certs from etc-ssl-certs (ro) /etc/tls/serving-cert from serving-cert (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wv2cn (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: etc-ssl-certs: Type: HostPath (bare host directory volume) Path: /etc/ssl/certs HostPathType: etc-cvo-updatepayloads: Type: HostPath (bare host directory volume) Path: /etc/cvo/updatepayloads HostPathType: serving-cert: Type: Secret (a volume populated by a Secret) SecretName: cluster-version-operator-serving-cert Optional: false kube-api-access-wv2cn: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: Burstable Node-Selectors: node-role.kubernetes.io/master= Tolerations: node-role.kubernetes.io/master:NoSchedule op=Exists node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/network-unavailable:NoSchedule op=Exists node.kubernetes.io/not-ready:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 120s node.kubernetes.io/unreachable:NoExecute op=Exists for 120s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 3h28m default-scheduler Successfully assigned openshift-cluster-version/cluster-version-operator-7cd8b7f756-5vrs5 to master-0.cluster1.savanna.lab.eng.rdu2.redhat.com Warning FailedMount 3h26m kubelet Unable to attach or mount volumes: unmounted volumes=[serving-cert], unattached volumes=[kube-api-access-wv2cn etc-ssl-certs etc-cvo-updatepayloads serving-cert]: timed out waiting for the condition Warning FailedMount 3h26m (x9 over 3h28m) kubelet MountVolume.SetUp failed for volume "serving-cert" : secret "cluster-version-operator-serving-cert" not found Normal Pulling 3h24m kubelet Pulling image "registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca" Normal Pulled 3h24m kubelet Successfully pulled image "registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca" in 3.194995787s Normal Created 3h24m kubelet Created container cluster-version-operator Normal Started 3h24m kubelet Started container cluster-version-operator Normal Pulled 178m kubelet Container image "registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca" already present on machine Normal Created 178m kubelet Created container cluster-version-operator Normal Started 178m kubelet Started container cluster-version-operator Warning Failed 172m kubelet Error: services have not yet been read at least once, cannot construct envvars Normal Pulled 172m (x2 over 172m) kubelet Container image "registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca" already present on machine Normal Created 172m kubelet Created container cluster-version-operator Normal Started 172m kubelet Started container cluster-version-operator Normal Pulled 163m kubelet Container image "registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca" already present on machine Normal Created 163m kubelet Created container cluster-version-operator Normal Started 163m kubelet Started container cluster-version-operator Normal Pulled 155m kubelet Container image "registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca" already present on machine Normal Created 155m kubelet Created container cluster-version-operator Normal Started 155m kubelet Started container cluster-version-operator Normal Pulled 117m kubelet Container image "registry.ran-vcl01.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:737b0a99451baef255890e60df61d5f50ccfc774c1be6cdf9aee08c56b3ac4ca" already present on machine Normal Created 117m kubelet Created container cluster-version-operator Normal Started 117m kubelet Started container cluster-version-operator
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759